I'd say it depends on the binder. If it's very bulky (which it probably isn't), then I'd say skip it for the visit. If it's the typical underworks or t-kingdom, then it should be fine.
I've listened to heartbeats and lung sounds through layers of clothing (not in actual clinical but in practice). If I can hear them clearly with my untrained ear, imagine what a doctor or NP with years of experience could hear through a thin binder.
As far as he or she being concerned about the binder itself, it depends on the doc. None of my mine have ever been, even when I brought it up specifically when I was having chest problems. I think if it's a commercially made binder and not an ace bandage, neoprene vest, or other DIY type, you'll be fine. Don't even bother bringing it up if they don't. If they do, you can either be completely honest or just say it's a compression shirt for fitness.*
Also, how would your mother find out about the binder if she didn't go back with you? I don't know how old you are (and please don't tell me/us), but if you're thinking about getting hormones, you should be at the age where your parents no longer have to go with you to the exam room. Just a thought.
ETA: *I realized that if this doctor is the one that will be prescribing you the hormones, they will most likely know what the binder is. So there will be no reason to lie or stretch the truth. If they have experience, then it shouldn't be anything new to them (and therefore won't ask about it) and they'll understand that a binder isn't exactly dangerous.