Guys I have known have had a variety of experiences. Mine dropped low pretty quickly, so, different from you. But my inflections and manner of speaking were fairly female, and still, sometimes, even with the low voice, AND THE NAME JOSEPH, I am ma'amed on the phone. I take this to mean it's about HOW i talk, both the content, and the more sing songy way that I think I talk, compared to men. For awhile I tried to copy men in a flattened vocal pattern, and I felt inauthentic. So, I am myself, and sometimes tell the customer service people. I am a man.
Usually saying: I am Joseph. I am a man. helps, but not always. At those times, I use it as spiritual practice in staying calm and not caring what others think wrongly about me.
Best to you.
Joseph