Not all non-heterosexuals are gender-non-conforming, mind you (in the broader sense of how we behave and present ourselves, rather than in the very specific sense of not comforming to society's expectation to be straight!). I know gay and bi men and women who are pretty "stereotypically straight" in terms of how they comfortably present themselves and just happen to fancy their own sex.
Also, not all trans people conform to the stereotypical image of their experienced gender. Some of us trans women, for example, are not interested in being feminine, but just in being female (I speak as someone who is basically a bi-ish dyke currently in the wrong body

). I assume it must be the same with some trans men too. And those of us who don't conform to the expectations of "normal trans" values can get just as much pressure to conform from the trans community as from the wider world, alas - just as gay/bi people can be under pressure from the gay community to dress and behave in more "stereotypically gay" ways.
It's up to us to conform to our own selves, in the end, even if that singles us out sometimes. There's always going to be group pressure both from the normative society at large, and from within any group with which we identify, beit gay, bi, trans, andro, or straight.