Even with a peri, nerve damage is possible.
I had a peri with Dr. Medalie, and in some ways I had the opposite problem that you are having. After my surgery, I was surprised by how much I felt the pain; I had read so many of accounts from guys (who had the DI) who said that they had very little pain, never really needed the vicodin, and felt pretty decent after only 2 weeks. It took 3 months for my chest to stop aching almost constantly, and it still acts up once in a while (about 18 months postop).
Another thing I noticed: my right side was slightly bigger than my left, so the surgeon had to cut out more tissue on my right side. Because of this, I was expecting my right side to hurt more, but it actually hurt less. When the bandages came off and the swelling went down, I found out why: despite it hurting like hell, I had almost full sensation to touch and pressure in my left side, but my right side was numb to touch and dulled to feeling pressure. Once my chest healed more, I tried pinching both my left and right side. My left felt pretty normal, but my right was hypersensitive and a light pinch hurt way more than it should. I took that as a good sign, actually; the nerves were still there.
After 18 months, I'd say my left side has almost normal sensation. My right side still has noticeable numb areas, though it has gotten heaps better. The hypersensitivity has largely (but not entirely) gone away, and I have gotten back a fair amount of touch/pressure sensation. Apparently nerves can grow, but it takes a really, really long time. I had some dental surgery done in middle school, and a large nerve in my gum line was severed. I had numb gums for a few years, but the sensation is back now. I'm hoping my chest has the same healing abilities, and it's getting there--slowly. Oddly enough, my areolae and nipples are still responsive to touch and temperature (cold or touch make the areolae shrink and the nipples protrude, with warmth having the opposite effect). Sorry if that's TMI, I just think it's neat that my nerves are still intact enough for that; I wasn't expecting it.
Hope some of this is useful to you, but really: 4 days is very early postop and with the swelling still going on, it's hard to tell. And even if you feel able to move your arms about, I'd recommend going easy on that. Even though they're smaller, peri scars can stretch, too.