I do get this, too. I used to think it'd be okay to be called both girl or boy, it didn't feel like a problem. Now, only half a year later, I feel strong disgust when someone calls me a girl and get all warm inside when people treat me like they would treat a boy or an androgyne.
I still identify as androgynous and feminine - just definitely not a girl/woman. My femininity comes almost completely from my boy side. It's totally possible that the real me just isn't a girl at all, and that was really just a shell.