They're all probably just used to it being your father when someone with a deep voice answers the phone, especially if you don't have any other men in your family. When they realise that you've got a deep voice too now, and it sinks into their subconsciousness then they'll start listening for other clues as to who here speaking to than just the pitch.