Brassard's instructions say that after 2 months, regular soap works to clean the vulva (and not to use *anything* inside the vagina, including douches, as it will disrupt the formation and maintenance of natural vaginal flora). Similarly, I've never used condoms, only washed my dilators in Hibiscrub for the first 2-3 months and regular antibacterial soap thereafter.
The "stuff" is probably dead skin, leftover lube, etc. shedding, and that will continue for a while. As long as there's no new odor or color and no discomfort, it's probably normal. (If you have sudden pain, itching, or a really bad smell, though, contact a [local] doctor; might be a minor infection.)