That type of thing happens to me sometimes, and I make sure when I'm going about my business to not make eye contact with strangers unless I need to. I'm not fulltime or anything - what is that anyway? - but do get quite a lot of stares when out in public because I keep a very androgynous look with shortish hair, very, very male clothes, but a feminine body and face.
You should never, ever shake a stranger's hand unless it's in a very public place or at work, and never, ever give anybody your phone number unless it's a secondary phone.
The thing about many men - especially gay, bisexual, and trans ->-bleeped-<-s who frequent the scene - is that they feel that because you are androgynous or trans, then you also frequent the scene and are therefore easy. What I mean by 'the scene' is places like bi sex clubs, adult cinemas, places like that. At those places it's normal for men to follow 'attractive' people around in the hopes that they might get sex out of them. Unfortunately, some people carry this with them into daily life, and feel that it's okay to approach or follow somebody who is clearly not part of the gender norm. I'm thinking in particular of that 30+ year old guy who shook your hand and pulled you in.
I have been to places like that in the past, and having people follow me around in that context is fine with me. Having it spill out into daily life though is creepy and scary and completely unacceptable though.
Only a few weeks ago for example, I was overseas and was visiting one of the main tourist attractions - a certain geothermal lagoon hotel. I'd just finished my dip in the lagoon, and was in the common area. It was quite early in the morning and there wasn't anybody else in the lagoon, and I thought the common area would be empty too, but there was a man sitting there just relaxing. I had to walk past him to get changed. As I walked past he spoke, saying "you know, you look really great." And I stopped, muttered a quick thank you, and trotted towards my locker to get my clothes and hide in the handicapped toilet to get dressed. He was waiting a little way outside the toilet after I'd got dressed, and I quickly stepped out of the place, went to my room to pack my things (it was my last day there) and headed to breakfast with my partner. He kept staring at us both during breakfast. When we went to leave to wait outside the hotel for the bus, he was standing near the hotel lobby window, milling around, and looking at me.
I was rather creeped out because he and I were by ourselves in the common area initially, and I think anything could have happened. He was a stocky, strong-looking man, and although I am not tiny - 5'8 and 150 lb - hormones have turned a lot of my weight into flab, and I'm nowhere near as strong as I used to be, and certainly can't defend myself against most men.
Again the safest thing to do is to not respond to strangers on the street, or respond only enough to be polite like a quick "Hello, sorry I'm in a rush got to go". If you want to meet people, meet them in other ways.
