Without wanting to be too much the smart arse the phrase that came to my head was "ask a silly question..."
Of course he's going to say that. Asking the question lets someone know - even if before they only suspected. Once they KNOW it is of course obvious - it has to be otherwise they are admitting that they cant tell the difference - which for a lot of people is embarrassing.
So you actually can't read too much into it.
I have observed that in social situations, where we are not known, although neither of us gets spotted often, on the very rare occasion that we are, I do better than Alison despite the fact that she is patently far more attractive than I am. The only real differences are voice, and the fact that while she adopts a traditional female presentation mine is deliberately a little tomboyish - tunic over jeans or something.
On occasion I have probed the reason why Alison was identified whilst I was not - two things come out time and again, 1. My voice sounds natural (which it is because I never had to change it thanks to a puberty failure) and 2. I look too deliberately tomboyish to be trans - because to quote one observer verbatim "no one who wanted to be perceived as female would deliberately choose to dress in the style that you do!" a bit of a barbed compliment there.
Number 1. is pure luck 2. is very deliberate inverse psychology. I learned very early on that the way that I could pass with as near 100% success as makes no difference was to look as though I was a woman TRYING to appear a bit masculine. So that's what I do. Simples!
Don't let it play on your mind Kate. You do very well. The bottom line is we can't control how other people see us. The best you can hope for, if it matters to you, is to learn, as I have I have done, how to stack the dice.