Without making a joke...
I don't see any reason why it could be bad. I mean, if you move around a lot in your sleep, the packer could shift and end up somewhere else (in the sheets, on the floor, etc). So that might damage your packer in some way, I suppose. But, in general, there shouldn't be a problem.