I'm not usually worried about things either, so its always a surprise when it happens. For me its when something is really important, which I'm sure this must be for you.
Assuming there's other psych's available in your area, you should just be able to find another one if this one's no good, or perhaps even find another endo or doctor who doesn't require two referrals.