I think this is one of those stupid things people say when they aren't sure what to say, so they go to the well of "dumb comments" that vaguely express what they really feel or believe.
An example is if someone close to you dies. People say all sorts of stupid stuff to that, most which reflects that it is an awkward situation and they don't know what to say. Usually the comments reflect whatever the person commenting believes (e.g., "She's in a better place now" = I believe in heaven/afterlife, not "wow, living on Earth sucks").
I think "It's a pity" remarks about being trans are not saying, "Well, if you were an ugly girl, I could understand you transitioning to be a boy" even if it sounds like that. I think the comments are supposed to be expressing solidarity with your Mom's grief. She has lost a daughter. Just like some people will say if you were dead, "Elijah was such a pretty girl" and they don't mean, "Wow, it's too bad your kid died, but if you had an ugly child, it would be ok." It's that it is such a shame that your Mom has to experience the grief of losing the daughter she had, and this is one way for people to express that sentiment in a way that meets our current (albeit transphobic) conventions.
When my Mom was upset that I cut my hair and said I had such pretty hair, it wasn't because if I had ugly hair, she'd have understood my transition. She was just upset and my hair represented that transition to her. Since girls are judged in our society so much by looks, it makes sense that people are focusing on the loss of that pretty "girl."
The odd thing from my perspective is that while our parents go through the mourning process for the child that isn't there anymore, we are not actually dead. As a result, a lot of the comments we get are the types of dumb things people would say if we had passed away, except now, we get to hear the stupid remarks directed towards us. I think when someone tells you that it is sad you transitioned for *whatever* reason (beauty or otherwise), the appropriate reaction is that it is sad that your family has to mourn the person they thought you were, but that it's a bittersweet time in that you hope that someday they will understand how happy you are and that this was a necessary step for your happiness.