A common concern, yes. Irrational, yes, as the odds are stacked well against the "never pass" scenario. As birkin mentioned, it takes some guys much longer before they are consistently read as male, but never getting there seems unheard of.
I'm self-conscious about my round skull shape and crap like that too. It's too easy to laser-point focus in on this stuff and blow it out of proportion. I cannot recall ever seeing a person that didn't have a few features that would look at home on a person of the opposite sex. Some guys have big lips, or a small frame, fat around the hips, high voice, etc. Some women have a square enough jawline to cause me envy, or big hands, 6 feet tall, deep voice, and so on. Often several such features are present. Being gendered correctly on sight is a matter of tipping the scale one way or another, not about having 100% stereotypically male or female features.