I agree you should just ask him about your concerns before getting too worried.
I don't get my testosterone levels checked. I'm pretty sure my doctor would order tests for that if I asked her to, or if she didn't want to, she'd explain why. If we had major differences of opinion on anything I would try to seek out another doctor.
I do get bloodwork somewhat regularly, metabolic panels and basic stuff to check how I'm doing. When I first started hormones I think it was every month or two, and as time has gone on it's down to every 4 to 6 months. My clinic put their system online in 2012, so I just looked at what's in the records, and there are a lot of edits where they "corrected" my gender. I'm not sure I want to care about how they are factoring that in. For most medical purposes, we are somewhere in between.
I worry more about my liver function and cholesterol and blood pressure than my T levels, personally. I would definitely raise hell about my T levels if I weren't experiencing changes though.