Yes, Brassard says it's possible to gain width simply by using larger dilators, as it's mostly a matter of stretching the PC muscle. I would imagine the limitation is actually the size of the opening in your pelvis.
However, you clearly would still need a larger dilator if you wanted to get up to his size before having sex with him. I will note, though, that the step up in circumference between the orange Soul Source dilator (4.7" circumference) and his 5.5" is approximately the same as the steps between dilators (for example, 1" to 1.25" is about 0.8" change in circumference), and most of us manage to do that with some discomfort and gentle stretching the first few times we go up a size. In other words, you probably *can* accommodate him just fine as long as he's patient and careful at first. Also, as Cherrie says, a penis is different - no matter how hard they are, it's not going to be as unforgiving as solid plastic. So a penis isn't actually going to stretch a vagina quite as much as its circumference would indicate, whereas the dilator does.