Heya Julia,
Even before I began my transition, I noticed that my humor wasn't always taken the same by men as by women. In discussing this with my wife, I discovered some differences in how humor tends to be delivered and received differently, generally, among normative males and females. Among men, there is a kind of "support" that's provided that can come off more like insults, like references to men being ugly, for example. Men tend to find this as a comfort, and groups are often socialized to give and receive this sort of "good-natured" ribbing. These same insults could feel hurtful and embarrassing coming from a woman.
Women, on the other hand, tend not to poke fun at each other in that way, so the "typically male humor," if you'll allow that just for the moment, can be mortifying when directed at a woman, even if by a woman. For example, whenever we're in a restaurant and someone drops a plate or a glass, my instinct is to immediately call attention to it, partly because I want to defuse any anxiety over it. In reality, if a woman is the one who drops it, she may tend to be far more embarrassed at the attention than a man. Of course, exceptions abound.
I imagine when we go through transition, it's possible that these kinds of cues get misrepresented and misinterpreted in ways that we might not intend or expect. The tease at your brother might very well have been taken differently if he sees you differently now. These are certainly broad generalizations, and don't reflect how everyone is, or how anyone should be. They're just observations that have informed and restrained some of my ideas about what's funny, and made me a little more careful.
Erin