If all of your paperwork matches, you should be fine. I believe the initial issue was the rule which allowed the use of 'X' for non-binary people. If your passport has 'SEX: X', then expect problems. If you never updated your birth certificate and have transitioned, when you apply for a passport it will have your assigned sex at birth. To the best of my knowledge, no state or federal authority has attempted to retroactively change updated birth certificates back to the original assigned sex.
I have two trans friends who recently traveled internationally, and they did not report any issues with their passports.
It's OK to be paranoid about this, but I think a trip to Canada would be fine.
Love always -- Jessica Rose