There's two sides to privilege; internal and external. External privilege is in how people deal with you. Internal privilege is how you interpret and react to the actions of others. When somebody says "It's all in your head," they're failing to understand the mechanisms behind internal privilege. We hear stories about women, for example, being taken less seriously, and see it happen with our own eyes. So, when we see evidence of that, we react to it. Sometimes it is purely up in our heads, but it's completely irrational to just accept that as a justification for every bit of evidence we see.
When our lived experience is denied to our faces, it makes is question our experiences. It's beyond frustrating, and makes it even harder to trust that people care, or that they're willing to listen. It makes us feel more alone, more invalidated, less worthy... and entirely reinforces the cultural norm. And then, we react -- we go along to get along, because it's just not worth a fight some days, especially when the evidence from yesterday shows that we won't be taken seriously in that fight.
However... one blindly act as though they have privilege. It can work. This is a superpower that transwomen can hone -- you were raised as a man... if you continue to act as though you have the privilege of a man, people might surprise you. Of course some of those surprises will be bad. But if you continue to act as though you have privilege, you're setting an example. Others will see that, and hopefully we can help establish a new norm.