generally babyface just means you have soft features and if people guess your age they'll tend to guess significantly low. I have this aaalll the time (I'm 32, in girl-mode I was generally getting people guessing 23-27 at most... last night I was out in as guy-mode as I could be and someone guessed 19-21, though interestingly didn't get carded where I would have as a girl)
As a guy, facial hair will go some way to hiding those younger features so that's probably what your friend was getting at.