It's not you're fault either, when you do keyhole surgery which I assume you had its always a possibility that your nipples will stretch a little. Because the skin on your areolas are more fragile (not sure if that's the right word but I think u understand what I mean) that happened to me too. My nipples are perhaps a little bigger than most guys but I don't really think about it and I never had any comments about it either. It's your brother that's prob why he's commented on it I don't think any strangers would give it a thought or anything.
I have 3 brothers and we pick on each other about everything, brothers always find something to pick on.