There's a guy at my LGBT center who is a little like the fellow you encountered. I'll be sitting in the lobby, waiting for a meeting to start, and he'll come up and start talking to me. I will let him ramble if I haven't got any work with me. I know he's not quite "right" and probably doesn't get much positive social contact.
If I have grading obligations that I must fulfill, I'll let the guy talk for a few minutes, and then I'll tell him that I brought work with me because I really have to get something done. He accepts this. But I give him some strokes first.
The guy you were dealing with wouldn't have been so easy to shake, obviously. I would have talked to him for a few minutes; then I would likely have found a good opportunity to kindly say, "Sorry, it's been nice talking, but I really came here to get some work done." If he kept pestering me, I would be firm but patient--to a point, and then I would mention him to one of the guys at the front desk, and let them deal with him. Does your center have staff like that?
Empathy is not a female trait, but I think a lot of straight guys have drunk the Flavor Aid and don't respond empathetically very often, at least not in any obvious way. Plenty of guys in my gay group are very empathetic, possibly because they have been through the wringer and have been able to throw off some of that early "male" conditioning. Or maybe they never had it to begin with.
There is nothing wrong with your being kind, especially considering your family experiences. But you seem to want to be more assertive. If that makes you happy, work on it. If not, keep going as you are. In my opinion, the world needs more gentle men. You just don't want to let too many people walk all over you because of it.
My therapist, who is gay, is the most empathetic man I know. Yet he's usually read as straight!