I guess it depends how deep it caves in, but I have a slight one myself, and it's actual less noticeable now after surgery, because the skin isn't being pulled down either side of it if that makes sense?
Equally though, one of my close friends has a more prominent one than me, and it doesn't look at all like breasts it purely looks like he has a dent in his chest, most people think it's pretty cool or just don't notice. If anything in T-shirts it just makes you look like you have bigger pecs.
So I wouldn't worry about it, you'll just look like a guy who has one!
Heres mine so you can see for comparison a bit maybe:
