Well, as someone who has experience in drawing, I have a theory that in part it has to do with the angle they're seeing your face from. I think our more masculine features tend to be more obvious from a lower viewpoint.
Also, it may be that you aren't actually passing to the adults, they just see someone who is more androgynous and can kinda/sorta tell, but have the common kindness not to say anything. Kids aren't as discreet.
Then again, one of the first people to not read me as male was this little kid at my school...he was like eight or ten, really sweet. I was seventeen, and he walked up behind me and asked me if I was a boy or a girl, because I was wearing this very androgynous purple sweater with knee-high boots. Of course, when a teacher told him "that's not a boy, it's a MAN!" it got less cute pretty fast, but still...kinds aren't as bound up in social convention. That's important to remember.