The only thing that really bothered me was not being able to say anything. When people pick up on something, and say something, and you have to, at worst deny it and, at best, not say anything.
My mum always used to say to me, jokingly, "If you were a woman, you'd be a b**ch!", and "God, put your claws away!" because I have a very... well... let's just say venomous, sarcastic tongue when I've a mind to. When someone ticks me off (which is extremely rare, but it does happen). And you have no idea how many times, when she said that, I wanted to say "But I am!"
Sadly, she passed away before I came out to anyone, so I never got the chance. But I wish deeply I had. I sometimes think she knew already, but I can't be sure.
Not just her. So many people have made throwaway comments about "You should have been born a woman, because you act like one." To which, at the time, I would just have to smile and shrug it off. But inside deeply wishing I had been brave enough, or sure enough to say something.
That's the main thing that annoyed me about being in the closet. The fact that you have to look through the gaps in the closet doors even when you want to open them wide. Even when people pick up on who you are without you saying anything. And you have to either deflect or deny. That's probably the most annoying thing of all.