I had to frog the whole entire thing and start over two sizes smaller. Didn't have to change needle size even though the first one was a big, sloppy looking huge (size-wise) mess. I think it's this yarn, Noro's Silk Garden. Tension is very important and I was trying to adjust mine.
This is looking to be much better. I'll do a few more inches on the body and then do the sleeves so I can try it on and adjust as I go.