I have exactly that same experience. A while back I was doing DIY HRT and after about a year, with minimal dysphoria, I stopped the HRT. As you said, the dysphoria came back, raging even worse than before.
So I don't really doubt that I'm trans despite this seeming contradiction. I also don't really know why it works this way for some of us. Sorry :-(.
What I'm doing though is continuing the HRT, growing my hair, and otherwise just living on a fine line right in the middle. For me at the present time this is the path of least resistance and it leaves my mental state several orders of magnitude better than before. Also, with the HRT changes so far I get addressed as ma'am often enough without putting forth any effort that it fills my need for affirmation.
Conform and be dull. —James Frank Dobie, The Voice of the Coyote