Unless you can realistically plan to live forever, then longevity probably isn't one's biggest concern - unless you think you'll be the kind of 100-year-old that can actually walk, run, and still appreciate rock and roll.
But even then... stopping HRT is not likely to be worthwhile for that sort of thing. I mean, the masculinizing thing is nasty - you can still pass with surgery and stuff, but there are some things that surgery can't do, and I doubt most elders could safely have surgeries for those things anyway.