As a mathematician, I am disinclined to draw a moral distinction between showing a "male" chest and a "female" chest. Society is rife with double-standards regarding gender, though, so it's only natural society gets hung up on non-binaries. The closest thing to a "clear" difference that emerges, I think, is for women to potentially produce noticeably-erect nipples -- indeed, women are often permitted to show about every part of their boobs except their nipples -- although I feel this point takes itself too seriously. Still, for someone getting on hormones, wearing a shirt around after three months may be good to avoid trouble (mileage varies).
KiraD's point was discussed fairly well in the film "This Film is Not Rated" (available on Netflix and elsewhere). Women's sexuality and "gay" sexuality tends to be rated more strictly by the MPAA than a "straight" male's sexuality* (even if it involves a male having sex with a pie). Given the structure of the MPAA (socially-conservative, secretive, unprofessional, and defensive) and the influence it has (it judges which behaviors are deemed "normal" by people who predominately watch PG-13 movies instead of R movies, or who watch R movies instead of NC-17 movies), it's not surprising that these attitudes are perpetuated. If women's sexuality was normalized a bit more, boobs would probably produce less of a reaction in people and may lead to a relaxation in policy (at least in some states) -- all the breastfeeding mothers out there would probably appreciate it.
* Granted, a lot of male sexuality involves female sexuality at the same time. However, the story/camera can place the focus on a person of one specific gender. When James Bond almost invariably has sex with an antagonist female, the focus is on *him*.
