Muscles do not make a man a man. Character does. So is true for a woman too.
If they continue to give you crap about muscles (and you're in the guy mode), you can always mention one of American founding fathers, Alexander Hamilton who advised for a sport to learn shooting guns

I normally tell them muscles would make my revolver draw slower and I am fine as I am, thank you.
However, some fitness, cardio, stretching exercises or self-defense might be good for your health and fun