Fear is a biological gift. You should not be trying to ignore it. If your body/mind is telling you to leave a situation, you should.
Do you need to interact with these people? Are they an integral part of your life that cannot be removed presently? I don't like to have people around that make me uncomfortable, and I've had a very easy transition partially because I'm willing to cut ties with just about anyone. It may mean the loss of a relationship, but it is a gain if it lessens your anxiety.