Working related to customer service, it's usually better to say nothing than to say something potentially offensive. I get 'you'd', 'you two'd' 'guys' (which is just gender neutral, when the same waiter says it to two teenage chicks at the next table), and I swear to god that people have invented the term 'ma'an' when they jumble words at me.
People often usually cut off very abruptly. In texas, man/sir/ma'am is basically punctuation, at least where I am. Where you would normally hear 'thank you sir' or 'thank you ma'am' it's just 'thank you' with an abrupt drop off/lingering silence and facial expression. Or as I said, half the time I think I hear 'ma'an'

Better to sound a little curt than to call out the wrong term.
Nobody's going to arrest you for being androgynous, look at the ridiculous teenage emo boys at your local mall. If you get cross-pronouned and you like it, and the person you're with doesn't burst into a fit of "LOL HE CALLED YOU LADY" or "excuse me, does my boyfriend look like a woman? don't answer that, ->-bleeped-<-, he's my boyfriend", then whatever.
For me it gets vely, vely awkwald when I'm not trying to pass and I pass. It's a feeling of 'Ah ->-bleeped-<-, time to pay attention.'