It really isn't hard on the body. Hormones are simply chemical messengers. Testosterone screaming over estrogen simply leaves estrogen unheard.
In females, to have no testosterone is a bad idea (I guess this applies to MtFs). You want a least a little in the body. I don't think there is any problem with males having no estrogen... although they'll always have some, even if it isn't much.
The removing on the ovaries may cause a change in your T dose. This is because the testosterone doesn't have to yell over the estrogen anymore. At least this is my impression, correct me if I'm wrong. Since we're transitioning to male, and if your female organs are dormant (T does that) then you don't have a need for estrogen, really...
Your body, however, will still have some in it. Not enough to make your female sexual characteristics return, however.