Like Alex said, depends on your age. But even if you're younger, I'd stay, shave it. The sideburn-fluff that ladies have doesn't grow like that on guys I believe. And I actually think it's same with the mustache, even for a guy that just started growing facial hair, the look is very different. Better make everybody think you've got a bit more, 'cause you're clean shaven from chin to ears than 'female-fluff' so to speak.
Edit:
Even if it's extra female-fluff. My boyfriend's body (still pre-T) has higher T-levels than an average woman, also meaning more facial hair and a bit thicker here and there, but he needs to shave it as well 'cause it's still too feminine.