That guy was definitely being weird. I think in this case you were right to trust your instincts.
A gentleman will never shout at you from a car window, he will not try to GET YOU INTO said car, he will not attempt to approach you as you're loading things into your own car (to grab your keys and shove you in the trunk???), and, most importantly, a gentleman will never ever push you on ANYTHING after you've said no. This guy was not a gentleman, he was a creep, and you did well to avoid him.