Having been in the sacristy in different parishes before Christmas and Paschal Triduum Masses and liturgies, it wouldn't surprise me if having to look for another lector was the real and only reason for his anger. Those days are really stressful, and anything not going as planned can be a huge source of frustration. That said, he shouldn't have yelled like that, especially knowing about your situation. I can attest that transition is a lot more stressful than even the Easter Vigil...
I can understand how you feel, though - I'm not going to Mass often myself, due to fear of reactions, be it disapproving looks from random parishioners, or reactions from people I know or priests. Still, I intellectually think it shouldn't stop me. I'd tell myself to go nevertheless. I just wish I could take my own advice.