It's not likely sperm was on the door. Most guys don't use the women's restroom...a few pervs might, and they might masturbate, but I wouldn't think they are neat and tidy enough to put a thin layer of semen on the handle, hoping some woman will get pregnant.
If they did, more than likely it was in a big glob, and you'd notice it. (Typical semen amount is like if you had a serious sinus infection, and blew your nose...big ol' gross palmful of stickiness).
Sperm also die pretty quickly when outside of a body, like within the hour. Also there's a saying, "If it dries, it dies." If you didn't feel any moisture on the door, you're safe.
There's nothing wrong with "doing yourself" in a public restroom btw...just be careful where you touch the facilities! (I flush with my foot, and use TP to open/close the stall door)
*hugs*