Spending a lifetime, totally forsaking your own sanity, pretending to be someone you are not, just so others are comfortable is the selfish part. Selfish on the part of the one calling you selfish.
Being true to your self is not being selfish, unless someone loses what they think they desire. They then are the true selfish one.