It's normal to be apprehensive about passing in public at first, but you've obviously been exposed to some very negative situations that have had a lasting effect on you. I don't know where those events occurred, but I suggest you avoid those places in the future. You need to seek out safe public venues and always go out with a friend or two. I've never experienced anything like you mentioned and I've been going out in heels and dressed since the '80s. But I admit, I'm very careful.
I suspect your friends are right about your passing well. Your height is enviable for a MTF person. Most of us are much taller. I'm 5' 11" shoeless, but I love the way I look in high heels. My legs are proportionally short so they visually lengthen my legs when I wear things like skinny jeans or pencil skirts. Still, I don't wear them often for the sake of walking comfort. I think the key is to always dress as others do where you are going. If it's a formal situation at which most women are wearing heels, you may stand out more by not wearing them.
Lyric