One thing to think about, hon, is who is in control of your life. It sounds from your posts that what you think other people will think of you, or say about you, puts them firmly in control of what you do and don't do. And your decisions are made around that. Kind of like "I have to do this because other people will think this or that, or they might say this or that."
One step you can take is to try and put yourself back in control of your life. If you do something or don't do something, make it because it's what you want. It's often easy to lose sight of why we do all this in the first place; namely to be able to live our lives the way we want to. If you put others in control of how you live then the changes may have been made but you're still not really living your own life, you know? Labels are things other people stick on you. To let go of it, you have to put yourself back in charge of who you are.
Letting people get close to you is largely a trust issue, and that can be very hard to overcome, speaking from personal experience. Two things I found which help are to make the effort to remember times where people were genuinely nice to you, and friendly, and didn't act the way you think they would. And coupled with that is to try to accept that you cannot control how other people think, and what they think. All you can control is how you yourself think, and how you live your life. Take it slowly, find common interests so you have a starting point, and take it from there. It's hard sometimes, I know, but if someone hates... that says more about them than it does about you. That doesn't mean everyone will be the same way. Sometimes you have to chip away quite a few chunks of rock before you get to the diamond in the rough.
Is it worth it? Yes, in my opinion.
*big hugs*