No, you do not need a packer. Use a packer only if lower dysphoria bugs you. I wore my packer 24/7 until I left for surgery, then ceremoniously tossed it into the trash.
Nobody is going to think anything; they shouldn't be looking at your crotch!! Secondly, post op I have no appreciable package. Stealing Jamie-O's statement: I'm hung like a housecat. I no longer pack and don't worry about my lack of bulge.
So, it's really up to you.
I *might* consider packing should I ever need to wear a speedo or something. But at my age that is highly unlikely!
Jay