My dad was an annoying jerk with a mean streak. I survived my childhood and you'll survive yours too. Just take his example as how to not act towards others.
On the bright side, him telling your brother to punch you in the face does show that he doesn't think of you as female, at all.