It's normal to see increasing androgenization after spiro withdrawal since spiro inhibits androgen action. Stopping it just because T levels were undetectable or "too low" is not right, in my opinion. Instead, the doctor should have asked you how you were feeling, determine your rate of feminization, assess your overall health to find out if any complications arose and if everything looked fine, then I see no reason to change anything. This change seems to have hurt you more than benefited you. My 2 cents...perhaps get a second opinion from another doctor?
They prioritized numbers over everything else, it seems, when your actual response to levels (how you feel, how your body is responding) should be the main determinant. If you felt good and responded well, then why change anything at all?