When I first bought stuff from the men's, I was worried that security guards or shop employees would find me out, laugh at me, and then throw me out because I was in the "wrong" section. They didn't. It doesn't really feel odd to me anymore, I mean, who cares what some random store employee might think of me? I am a young man, regardless of wether they see me as one or not. One's worth is not in the eyes of others.
I get the same feeling when going to the women's underwear department, like I'm doing something very forbidden, trespassing or something. This feeling has always been here, the "Oh, I shouldn't be here, what if they find me out?!"