Joey, maybe more layering? Instead of a pull over sweatshirt, get a zippered one (thrift store!). Then you have a t-shirt and sweat shirt. I am the prince of layers. I wear button down shirts which actually might help, but might look a bit older for you. I put on a muscle shirt, then a tshirt, and then a button down shirt.
I think you have the misfortune of a bigger chest. Yikes must be bad.
---Jay