If you're going to visit Australia during one of our summers, it would be stupid NOT to wear a shirt of some kind. It's summer here now and we've just experiences the worst heat wave in 153 years of recording (30+ degrees celcius, right up to low 40's), there are warnings on TV about the possibility of dying if you're not careful, so wearing a shirt wouldn't even get you a second glance.
However, if you're coming in winter, a proper wetsuit would be much more advisible. Despite what people think, as hot as our summers can be, our winters can be just as chilly depending on where you are (where I am I've seen it drop bellow -10 degrees celcius), and the water can get icy this close to the antarctic.
I think you should really just be yourself, the people here are pretty accepting, and I don't think you should have any problem being seen for you who are