I sidestep this issue altogether and keep my shirt on around others. But here's how I've handled other (not necessarily trans-related) questions in the past, that I didn't want to answer:
"That's a personal question."
Said matter-of-factly as you can manage. If you want to soften it, include a polite smile and slight furrowing of the eyebrows to indicate discomfort.
I have yet to have someone not drop it immediately. Yes, they're uncomfortable. But the moment they ask me a question for no good reason, which I don't want to answer, I have two choices. 1) Share information I'm uncomfortable sharing or 2) Let them be uncomfortable and keep my business to myself. I value my privacy more than their unbridled curiosity.