I identify as straight, personally I'm fine with the association.
The way I see it, we are all connected in the sense that we are all persecuted because we don't conform to the expectations of the gender other people have assigned us.
In that sense, we are similar.
I find I can get along with LGBT people pretty well because even if we have very different problems, at the core it still boils down to the same thing, which is that we are not like the majority and are often ridiculed or persecuted for it.
Doesn't mean we all have to agree with eachother all of the time.