I don't take it to mean "screw them" as in "get them" or "pay them back"
i certainly wouldn't agree with it if that were the meaning
I took it as "forget them" or "disregard them"
In that context, I wholeheartedly agree. Outside the three people who share my home, my take is "you are entitled to your opinion, you are not entitled to inflict it upon me" - in other words, if you dislike or disapprove of me...oh well.