Unless they are marking, cats really don't take up that much space. If you aren't sure if you would be sending the cats back to a good situation, I wouldn't do it. Its hard enough if they are going to a really good place, but if you aren't sure, board the guilt train.
Not knowing your boyfriend, I can't say for sure, but if they are his cats, getting rid of them may cause some tension.
When I moved in with my friends after a nasty divorce, I brought with me an African Gray parrot, two ferrets, a poodle and a cat. They already had a dog and three cats plus a cockatoo. Somehow we made it work. You could try it and see how things work out if you aren't sure how well the ex is going to care for them.
sam1234