I don't think it unusual to shift focus to the lack of a penis once the breasts are gone. Breasts are a huge issue, so not having the correct trouser contents doesn't get as much brain time. Now that brain time has been freed up, so it moves on to the other issue.
I think I had a drop in sex drive after my chest surgery come to think about it. I was so worn out from life which was hectic (doing school terms and co-op terms alternately), and the stress of getting and recovering from surgery, that I just wasn't in the mood.
Do you do weight training? I have found that working on the rest of my body to make it buff and sexy, can help dull the obsession on not having a penis. It also can help rejuvenate sex drive.
If things continue to be rocky, there is always couples counselling. If anything it shows your significant other you want things to work.
Good luck sir,
Shortman