On just a soap bar (wait, you wash by rubbing a soap bar against yourself? Never heard of that), it's a bit weird, but I've had that in a way. The first time I used my new exfoliating glove, some hair on my legs did come out in balls of hair still attached to a few living hairs, that I had to cut.
I don't think it's worrysome. I've never heard of fast body hair loss being a dangerous symptom. Unless you start noticing something else that's really dangerous-looking, I think it's just the anti-androgen working well. It never hurts to get a health check-up, but I wouldn't be in a huge hurry to have one done just because of that. Though maybe it's just thinning the hair and the old large ones are falling off, not supported because the root became smaller (imagine attaching a log on a vine). So this might mean 1. it will come back, but not as bad as before, 2. there's a decent chance of getting ingrowns, so exfoliate well.