If you ask me, the whole concept of anyone's chest being "inappropriate" is complete bunk. And the law agrees, since basically declaring boobs indecent exposure while bare male chests aren't is pretty discriminatory. (Women have won cases on the right to be topless in public.)
Really, it's just an issue of boobs being seen as sexualized objects because of our male-centered society. So basically, if guys could be turned on by it, it's considered obscene, if not, it's not. I call shenanigans on that, but whatever.
If you want a straight answer, I'd say that it's somewhere between the Tanner III and Tanner IV stage is where it would probably be seen as inappropriate, since that's the stage where soft tissue starts growing in and they start taking on an actual round boob shape rather than the flat limp moob look.