Yep, definitely sounds like your husband has some stuff that he needs to work through with a gender therapist.
From your side of things, the best thing you can do is be supportive, ask questions, express your concerns and recognise that, in sharing this with you, he's put a lot of trust in you. Internalising things and feeling bad for having a vagina isn't going to help you, him or the situation....it's not your fault that you were both born with the genitals you were born with; those sort of thoughts are most likely more symptomatic of the massive thing that has been shared with you and your brain trying to make sense of it.
If you're wanting to stay with him then the absolute best thing you can do is be supportive and give him the space to figure out what's going on. Don't make it about you; whilst it does effect you, massively, and there will be issues that are thrown up as a result of this which you'll want to deal with that your husband will need to support you through, his genital dysphoria isn't anything you are responsible for or should feel guilty for.
It sounds like you're having a rough time, I'd imagine your husband is as well, but the main thing to take away from this is that he trusts you enough to confide this in you and your relationship sounds like it is strong enough so that you feel confident in supporting him through it.
Obviously, I'm not a therapist and I dont know everything about your situation; see a professional as soon as possible, as a couple and separately