The one major compromise my wife requested of me in my entire transition was that I keep my old voice at home for her, and as it didn't bother me (and I owed her *so* much for otherwise going along with everything I needed), I did so.
I've noticed that somewhere in the last year or so even that home voice has mostly lost the male resonance and inflections, and I just can't do it anymore, but I do still relax back into my natural lower pitch. Part of it is, admittedly, that I'm lucky enough to have a naturally higher pitched voice anyway; still light tenor, but it's not as dramatic for my vocal chords as going back to bass would be, I imagine.
I never had a problem switching back and forth, and around other people I even scream/yelp/sneeze in my female voice, so it hasn't caused any issues for me. I look at it now more or less the way lxnder does, that when I'm alone or with my wife I can relax a lot more. (My wife also doesn't put as much effort into sounding "animated" and therefore feminine at home as she does in public, either; we're both introverts who like to stop performing at home, what can I say.)