I've never been one to chat in the restroom either, unless I'm with someone I know, and we're continuing the conversation that started outside. I've always been a little baffled by people who strike up a conversation as we're washing hands. (Or worse, while you're in the stalls. Really, unless the question is "Hey, do you have any toilet paper over there? There's none over here," there is no excuse for starting a conversation over the toilet.) I usually just smile and make pleasant, noises as seems appropriate at the time (Ah? Mmm-hmm. *polite laugh*) then try to make my escape as quickly as possible.
I find it interesting that while the ladies seem to be split on the issue, the guys all seem to be non-talkers by nature.