FWIW, I'm going the other way, but I've never been comfortable baring my chest, and have usually worn a T-shirt in the past, so I don't think people would think you're weird (for a guy) for wearing something that covers your upper body. (Okay, some guys will, but then some guys will use any excuse to put another guy down, it's how they "prove their masculinity".)
As for pool rules, I believe there are (men's) shirts specifically made for swimming, and I would think they'd get around any pool rule about not wearing non-swim clothes into the pool. So I would think a swim binder (which I assume just looks to the uninitiated like a shirt) would be treated the same.
I admire your courage. I'm going to start dressing as a woman for swimming this summer (i.e., wearing a swim dress and cap), and I'm terrified that I'm going to get thrown out of places for it.
ETA: I'm not saying you're in any actual danger of getting hassled or should need any courage, it's just that I personally am always convinced I'm going to be and always have to repeat my "it is a good day to die!" mantra over and over again to get the courage just to do the simplest TG-ish things. I'll probably be 90 (28 years from now) before I get the guts to go into the ladies' locker room.