Love being hugged by the men in my gay group. Don't like hugging women. I do it occasionally with very very special women...but, come to think of it, only two, and they're both trans. And I haven't seen either of them in months.
My family was very hands off, and I especially hated being touched by my mother, but I had (still have) serious issues with her. When my school called her and told her I had tried to kill myself (I hadn't), she came in all aflutter and hugged me. I still think of that as one of the most disturbing and disgusting moments of my life.
But then, as I said, I had issues with her.
Apparently, your issue isn't with your mom but with women in general. Could you be distancing yourself from women because you don't identify as one?