I had something similar but different happen to me. I was on my way back from seeing my therapist about a week ago, and she's kinda far away but the drive isn't too bad. I stopped at a rest area to get a soda and noticed the car that followed behind me parked right next to me at the end of an otherwise empty parking lot, about as far from the rest area building as you can get (I'm trying to squeeze in extra walking everywhere I can, not that it does that much, but anyway...)
I thought that was pretty weird, but I went and got what I wanted, then noticed the guy in the car was staring at me the whole time I was walking back. I started to get a little scared, so I got in my car quickly and locked the door. I cleaned up a few things in my car thinking by the time I was done he would be gone or something because I was obviously just being paranoid. Nope - I look up and the guy's eyes are locked on me still. As soon as he saw me start my car, he took off.
So then I did something potentially stupid and followed him, but he got off the very next exit, which was all of a half mile down the road. The whole thing really freaked me out.