It's all about what you feel comfortable doing.
Personally, I don't give a rats behind about going in the mens room. I just go in do my business and get out. The only time I don't use the mens restroom is during school which is because I don't use those restrooms.
From my experience men don't care, they just go in and do their stuff. It's not a chatroom or a social hangout if anything you will get a glance. I think the only way someone would really look hard at you is if it is a homosexual male checking you out, someone that you know (ie old friend), or you REALLY don't pass - as in you are wearing bright red lipstick, chest flying out, eye shadow, pink nails, and long hair with a headband.