Shaving tends to irritate my already sensitive face and make me break out, so I'm glad that facial hair is finally coming in well enough that I can stand to just trim it once a week or so. To each their own! I looooved shaving before it reached this point though, so I can definitely see the appeal.
I don't think facial hair implies passing at all, and I'm sorry that folks have said as much to you. I've seen even the most masculine guys will full beards and years on T get misgendered randomly, so having a beard isn't a guarantee that someone won't randomly use the wrong pronouns. I think it's much more down to face shape than it is facial hair, especially before they hear you speak. And that's just all letting T work its magic over a period of time.