I'm going to say it might be a confidence thing as well. Or just how you carry yourself. You look male, and I've always thought so. And your voice I think is in range. Being short isn't always a deal breaker. I see plenty of pretty short guys now that I actually pay attention.
But if you think about it, not many other guys would put up with something like that - they'd probably give the other guy a weird look, and say something like, "I'm not female".
Also, we all can't see how you walk or gestures or anything like that - especially in a candid, out-in-public situation (you could always take a vid but that wouldn't be candid).