Depth. Brassard also used to tell us that width could be gained anytime but depth once lost was gone forever; the instructions also used to say that if you couldn't manage a bigger dilator, to do exactly what you're doing - maintain depth with more frequent dilations with the smaller one until you can work up to bigger.
My own experience was that when I went on a business trip and brought only the smallest dilator (no room in the suitcase), the bigger ones were a bit painful and stretching upon my return, but I was back up to the orange one within 2-3 days. So it was definitely true for me.
Also, as Cosmo will tell you...

If you're having sex with men, there are tips and tricks for dealing with a really long penis, but a too-wide one just isn't going in, period. So I guess from a sex standpoint, width matters more (but again, since any loss there is temporary, it's not as crucial to *maintain* it).