When I used to attend church, females were allowed to wear hats inside. Males were not. It was about courtesy and respect. In the South, it is common for men to remove their hats in front of women, for the same reason. That could explain why your dad doesn't make a fuss about you wearing a hat, but will jump all over your brother about it.