You did the correct thing by walking away that situation already seemed to be getting alarming
its tempting to not say something back but the thing is you didn't and that's good.
Next you're not ugly don't let people like that push you down sure he figured it out but that doesn't make you ugly and who is to say he wouldn't have done this to anyone else?