it could be a Sebaceous cyst.. I heard somewhere that 'excessive' testosterone or swollen hair follicles can cause them.. They're pretty much exactly what you're describing.
Try putting heat on it for a few minutes a day for a week or so, that can help it 'dissolve'.
If not then, yeah, probably get it checked.