I completely get your frustration. I don't know you well enough to even know the details of your transition, and so many factors come into play.
Some guys do take longer. As far as I can tell, very few of us who are on T for a long time ever get stuck in a no man's land (urgh, unintentionally funny--or not) of androgyny. Three years is a while if you have been on a strong dose for the whole period. But only time will tell.
Also, are these the only people questioning your status? Is it possible that your young man is inadvertently dropping hints or signals to his closest friends?
Frankly, I would want to know as much as possible about my "tells." Knowledge is power--or can be--even if the knowledge is unpleasant.
P.S. Some people find that not caring how they come across to others makes a difference. Maybe "embracing androgyny" really IS the ticket.