Hi Valerie (what a beautiful name, by the way!),
I understand that this situation pissed you off.
I don't know if this helps you, but my mom is the kind of extroverted person who talks before she thinks. Since I was a kid - and long before coming out, mind you - she sometimes accidentally used my father's name instead of mine when she addressed me (like: Ted, er, Cathrin, would you like some more tea?, or even better: "TedCathryn..."). She sometimes also mixed up other people's names, even of her family. But that's my mom.
There is also an annual transsexuals/transgender meeting, the biggest of Germany, which I attend. Takes a whole week-end, it is awesome, but at the meeting last year, after two days, I made a pronoun slip and immediately excused saying: "Well, I've been at this meeting for two days, lots of new impressions in a short time, my brain is short from exploding, and it gets confusing here after a while..." Everybody laughed, and the person who I had addressed wrongly laughed as well.
I studied linguistics and had a course about slips of the tongue. The fact that you get to deal with accidental pronouns only every couple of months is a good sign that the colleagues around you really see you as a woman, otherwise they would happen more often. There are two possibilities how these pronoun slips of the tongue may happen for transsexuals - or people with a transsexual past:
1. These colleagues are somewhat aware of your "male mode" past, at least occasionally and subconsciously, so this slip of the tongue happens. But it seems to be rarely in their minds as you only get addressed with male pronouns rarely. Congrats! The same kind of things may happen to people who got married and are not Mr. Brown any more, but Mr. Smith now.
2. The same reason why other certain slips of the tongue happen. "He" and "she" are two opposites and closely linked in the mind. When a person speaks, they already start building the next sentences in their mind, certain words and the concepts related to them are activated in the brain network. Usually, the strongest activation is on the "target word", but it may happen that related words are also activated in your brain. If something goes wrong during computing, you get a slip of the tongue, for example saying "green" instead of "blue" as both are activated and both are colors, though "blue" has a stronger activation. It is rather unusual, in contrast, to say "Boston" or "agreeable" instead of "blue" as there is no/almost no simultaneous activation.
The same happens with "he" and "she", both are strongly related, grammatically interchangeable, the only difference is the gender. This is probably the reason why you had the slip of the tongue with the latin guy at the pool.
From my perspective, I'd say that you're not just tolerated, but completely accepted with your female identity at work, and regarded as female. Slips of the tongue would be much more frequent otherwise. Most of the slips you get are residues of a past long time ago...