Many doctors will write a referral without an office visit, if they take your word for it that you need a specialist. Basically, you're asking them to punt to an expert so they don't have to deal with something they know nothing about, and they're frequently happy to do so.

Referrals are something the doctor's offices will be VERY familiar with, and it's no trouble for them to produce one.
(I have gotten referrals for MRIs, skin mole checks, and eye exams without needing to see the GP; I call and say "I've got X issue and want to go see Y specialist," and the doctor produces a standard referral form and faxes it over to the specialist.)