Yeah, mine's like that too. My therapist was the one who broke the news to me actually and she pretty much said that to change my gender and name on everything else, I had to find a state that didn't require the presentation of a birth certificate when getting a new drivers license. Such as Rhode Island. Establish a brief residency there and change my gender, then move to a different state and change my name.
All of that just because my home state won't change the gender.
After that, changing your passport should be easy. I think they just need another form of ID, so you could just use the license you so painstakingly changed.
This is such a bitch. My jaw literally dropped when she told me all of this.