MeTony and I are some of the lucky ones, I'm not going to lie - I get sir'd from behind decently often at work while I'm pre-everything and don't even wear compressive bras. Looking at my legs, you'd think I was my father.
Passing it part attire, part voice, and part physical characteristics, but a lot of it is also bodily movement, and language. Do you say "excuse me, sir" or do you say "'scuse me, boss"? "That's alright", or "no worries, you're good"? W/r/t movement, do you take up space? Do you move your shoulders/back/arms more than your hips? Do you walk with your knees or your feet? What are your hands doing when they're idle? Do you touch your face or hair a lot? Men tend to move "un-self-consciously", while women move more "effortlessly". Are your movements tightly monitored when you're doing something physical? If so, stop!
If you haven't tried this, go people-watching. Grab a coffee at starbucks or something, sit down for 15 minutes and watch customers, pay attention to what the men are doing, how they're standing, where they're looking if their face isn't buried in their phone, how they set their shoulders, etc. For some of us, we've picked up on these cues naturally from an early age (and some of us went through a lot of heartache to unlearn them while we tried being women for a while), and for some of us, it's gotta be a conscious effort.
Also, you might seek out the help of a voice therapist. The one thing I can say for certain, getting chummy with the men at my work far more often than the women, is that men generally speak "sloppy", and very close to vocal fry the vast majority of the time. I also read someplace that how men and women pronounce "s" is also a kind of an unconscious clue to most ears - women, I believe, tend to produce a much sharper "s" sound at the front of the mouth, while men seem to do theirs closer to the soft palate, or flat-out voice it to the point of sounding closer to a "z".