It took about a month for me to regain most of my mobility. No real pain, just stiffness in the muscles.
I would personally recommend wearing that thing for as long as your surgeon wants you to. It helps shape your chest and guards against complications. 4 days in is very early. Your nipples may not be completely loose, they need to re-grow to the surrounding skin, and complications can still occur.
I wore the compression garment happily after I had several complications, knowing that it potentially safeguarded me from further trouble.
Perhaps you can ask your surgeon if you can loosen it a bit. Play it safe, man.