You're not Caitlyn, not a part of the Kardashian media circus.
I expect fully half of the disparagement owes to the Kardashian tribal hoopla. Your work place may very well afford a different perspective when/if you decide to transition. Transition can take years. Done well, the progression is glacial. Gives people time to adjust.