In the manly thought process, nipples are a womanly thing, and chests are to be looked at and admired, rather than touched. Only women may touch and only with the intention of enhancing the manly feeling.
This is completely unhelpful in views of the actual topic but just my 2c.