To a certain extent, yes.
It is like the people around you expect this from you. After all, you 'want' to be a man, so you should act like one. And if you don't act like a stereotypical guy, then your 'desire' to be a man is not real. Of course, this is utter bullfeces. Yet it seems to happen a lot.
As an example, I'm wearing pyamas that I've had for a decade now. They are the soft, girly kind. The boyfriend of my mother once commented that now that I was 'turning' into a man, I should get rid of them. Like the definition of manliness or being a man is based on what you sleep in. It's ridiculous. I'll wear what I want to wear, thank you very much.
So really, don't mind these people. They are just blabbering about what was drilled into them by society. They don't know any better. Keep on being confident, I say! That's the right route to take.