Personally, I think, the damage has been done. He may not express the way he feels about you, but being in the presence of a person who thinks of you in derogatory way is not worth your time. Either hang out with your friends without him, and if they refuse, and prefer him, then you know what you do. Good luck!