Ok, so as I understand it, you're worried about the social consequences of changes in your body shape when you're in the male locker room?
I might be biased because I'm probably more libertine than most, but I wouldn't sweat it, honestly. Don't let it bother you at all. Think about it this way.
There is one human drive which tends to outweigh almost all other concerns. It has been demonstrated to be more powerful, in most cases, than human compassion or even desire for self-preservation in cases of ambiguous danger. I'm talking about the drive to conform to the expectations of a social situation. This is profoundly powerful, doubly so in cases where the circumstances are unclear.
So yeah, you could have a significantly feminized male body, and I think almost all people would simply go with the expected behavior of walking in, doing their thing in a business-like manner without paying excess attention to the men next to them, and walking out.
Beyond that, breast growth isn't any kind of huge red flag. In most gyms in the US, you have your share of men with high levels of body fat who have developed moderate pseudogynecomastia as a result. The incidence of this condition is only increasing internationally as levels of obesity rise across the developed world. Meanwhile, there are also those with low body fat who may develop this due to glandular conditions. People may not know the details, but I think most have some understanding that these things occur and don't give others trouble over it. Heck, if larger than ideal male breast growth really caused people to react negatively, we'd have daily riots in every 24 Hour Fitness in Mississippi.
I do kinda sympathize. I've got a gym membership, and I very often hop on over to go swimming at night. I mostly look male, but I'm closer to average female than average male when you look at some of my body metrics. Nobody really notices or cares. I did have a guy staring at me yesterday. But I've been with enough guys to tell that A. he was gay and B. it was the "now take off the rest" stare.
On the off-chance you do encounter the one outlier (AKA ->-bleeped-<-) who comments negatively about it, ignore him. Honestly? That kind of person is bad to a lot of people regardless of their gender, and it makes no sense to adjust one's behavior to appease this kind of ignorant loudmouth.