i think it has to do with people not really expecting to see trans people everywhere, so they're not conscious of the little signs. when you're alone, they don't expect anything other than a woman, so they overlook those signs. when you're with a friend who doesn't pass well, they're reminded that trans people exist, and will study you to see if you are too. then of course, they'll notice those little things that are kind of different.
you'll have to take care of yourself. if it's devastating to be clocked, that's a good reason to avoid it.
but if your friend is worth more to you, and you have enough strength to deal with it, then it would be odd if you don't go out with her...
pretty much everything in life is a matter of priorities.
it's not mean to be true to your own feelings. "abandoning" your friend may hurt her, but that hurt isn't necessarily greater than your hurt if you have to force yourself into triggering situations for her sake. one would be hurting for the other's sake either way.