I was also surrounded by feminism but I've since begun to educate myself on what constitutes both male and female privilege in human society, in an attempt to find a much more objective appraisal of mens' and womens' place in the world.
The idea we have at the moment that being a man is somehow easier, somehow safer, somehow more rewarding, I've found to be something of a myth. I can see with my own eyes that being a man is different from being a woman and that women have many privileges as well, such the privilege of people being more concerned about a woman's safety and well-being than a man's, people not expecting you to do extremely physical and demanding work as a woman, being allowed to express your feelings and fears publicly as a woman without as much ridicule, being allowed to express gender fluidity with less social repercussions than a man, not being demanded that you potentially sacrifice your life should a war break out, now in the developed world having complete bodily autonomy when it comes to fertility and childbirth if you choose to, not being as severely punished in most cases should you commit the exact same crime as a man, being able to physically assault men with less social repercussions than if a man does the same to you, being less likely to be violently attacked by strangers (the statistics prove it) than a man, more than likely to be rescued from a burning building before a man is... And so on and so on. There are many many examples of how being a woman is potentially better or easier than being a man, depending on what the desirable criteria is.
Transmen themselves express their fears of what it means to be a male in society eloquently - am I going to be physically beat on more in a fight now? Am I more at risk on the streets? Am I going to have to be very careful around women & children now or else be thought of as a creepy weirdo? Am I going to be excluded from women's circles and become more lonely now? Will I have to compete with men to be respected now? Will my worth as a man be measured by what I do and how much money I make? etc.
Now that I am aware of this, I feel not that I am passing into some golden privileged domain, but that I am passing into a domain where I'm really going to have to watch my step because it is far less forgiving than the world was for me as a female. As such, I feel more ashamed of the many privileges I had before as a female than the ones I may gain as a male, because I did not have to do a thing to earn them as a female - I just had to exist. I feel quite ashamed of my ignorance during that time to be honest, because my female privilege was rather invisible to me at the time, and I had women telling me men are pigs while those same men were out at work all day so their spouses could smack talk them behind their backs in the nice houses that work afforded. So people who may point at and accuse you of seeking to gain some quasi-mythical superpower just by virtue of being male are only telling you - or only aware - of half the story.
It would certainly make more rational sense to remain being considered a female, if I wanted a less strenuous, less lonely, and safer life, I think. The condition of being trans isn't an issue of choice, though. It is what it is, and if it means greater hardship and responsibility as a man then there's nothing I can do about that. I will only feel ashamed of myself from now on if I do not face what it is that I have to do as a man in the way I feel I ought to. So the shame is gone, thanks to the extra information I learned over time about how it's not really men vs women, but that life is harder or easier for both sexes, but in different ways. Hateful people out there have sought to ignite some kind of war between men and women, and I really think some of them are setting us back, not moving us forward. Especially those who constantly claim that women are victims of everything, women are only ever acted upon and cannot act for themselves and not take responsibility for themselves and their choices. It can't ever be empowering to be a female and constantly hear that kind of nonsense.
There is indeed a lot of phallic symbolism in society, but equally there is a great deal of feminine symbolism and the power of female sexuality cannot be denied. Actually I would think the latter is more powerful. Although a man is supposed to have a penis according to society, he cannot get what he wants in life just by having a penis; he also has to prove himself worthy in other ways too. I do try to forget what the bodypart itself means or what its value is to an individual person, but then I ask myself how many people are going to potentially want me as a sexual partner without one? And are people going to value me less as a result if they find out about it? I fear almost certainly. Still, there's far more to human life than penises and being of sexual value to another person. The way my cards have been dealt and my castrated reality has meant that I was always going to have to find other ways in which to be of value and to value myself. I haven't really found it difficult to stop worrying for the future about the lack of penis, but that is mainly due to my low sex drive, and the fact I don't tend to think penis = "man". I know if it were higher, I might have much more serious issue with it.