I don't know how much this will help, especially coming from an MtF, but what you describe sounds like my life right before I started HRT.
I never developed any male secondary sex characteristics. I had a buzz cut, visible ink, all guy clothes, combat boots, really working to be seen as a man, and I just looked like a butch woman. I even had other guys tell me I was in the wrong bathroom. I was over 40 at the time, by the way.
What I'm getting at is there are cis men that deal with this too just as there are cis women that deal with the anti-transwomen stuff. I know it's hard to not quite fit in a box, but you have to live for you, not for what society expects of you.
That being said, I would second mannerisms. You're probably giving tells that you don't even notice. I'd also second voice training. Even men that aren't monotone in speaking, are compared to women.