I'm 153 cm and a midget amongst giants here in Dutchieland. Honestly, the average man is 187 at least. I still pass without a doubt and I'm not even confident.
So yeah, it's not all about size, thankfully. People just think 'wow, that's a small dude' not 'that must have been a woman at some point because he is so small'.