Yeah...it's not a game. There's a lot more to being seen as a guy than just being called "sir" every now and then.
Just as the ladies, when they start to present, are exposing themselves to a greatly increased risk of rape, sexual assault, and harassment, FTMs and passing FTAs are exposing ourselves to a greatly increased risk of non-sexual violence.
The good news is that if you don't intend to physically transition, you can probably get away with defending yourself without legal consequences (a cis guy or legally-male FTM who defends himself in a fight runs a fairly high chance of being arrested and charged himself, but women are generally treated as victims).
The bad news is that if you don't physically transition, you're likely going to be at a fairly severe disadvantage in the self-defense department.
The worse news is that if they find out what's under your clothes, you get the two-for-one deal: physical assault as a man followed up with rape as a woman.
