Gotcha. I've never worn one that didn't have something between or around the legs for more support. I've also never worn one that wasn't entirely made of stretchable fabric. I typically err on the side of larger, but it really depends how you're going to wear it.
Most guys wear the harness underneath underwear. If you wear boxers or go commando, I'd say go with a small. Boxer briefs or briefs, medium. That way it's comfortable but secure in both scenarios.