I would think you'd want it to get checked out. My first guess would be some sort of androgen insensitivity (and maybe the elevated levels would come from it not getting taken up and excreted or converted? But there are a number of conditions I'm aware of, and probably even more that I'm not aware of that could lead to this?
As you point out, the fact that you don't/didn't respond to those levels seems like one reason the caregivers at the time might have ignored it, or assumed perhaps you were self-medicating? I'm sure the diff Dx is more complicated than I'm personally aware of. Maybe others will comment on this... I have read some studies that have suggested, for at least some of us, that androgen insensitivity is something some of us happen to be "blessed with" among those who have been studied most closely? Easy to jumble this stuff up, and docs have a tendency to ignore lab values that don't seem to show any physical signs, unless they're something that can go silent but lead to major problems down the road.
If they were assuming you were on track for orchiectomy and SRS at some point, they might just have seen it as a curiosity to note, but not as something they felt they needed to discuss with you in depth, if it was clinically of no consequence?