I think you've done the right thing. And I know what it is to be in role and have to avoid one confrontation out of fear that a reading may make things worse.
However, that guy has issues. Not with you personally, but he seems to me to be a psycho (you said "he was yelling at everyone"). I wouldn't worry much, but I would be careful: such psycho guys may become predatory with people they might see as "weak" for refusing a confrontation and minding their own business. and since this guy seems to be a janitor at your workplace, you may be seeing him around. If he again gives you some headaches, he might be harassing you, in which case I think you just should report him to the right person (you'll know since it's your workplace). Regardless of the motivation (not even gender related, although the restroom thing might become one), I think it's best to never ever get into a fight at work, not even verbally. If someone predates you, just play the offended part (as it is true) and report. Well, I am basing my opinion on conflicts I've seen at former workplaces in which T* was of no importance, so mine may not be the best tip....
Keep well man!