It's definitely a complicated situation. On the one hand, you have a right to be respected in your own home. On the other, parents have a right to raise their children as they see fit and protect them from being exposed to things they don't feel are appropriate.
Yes, I do think calling it a nickname is a mistake, since in most cultures nicknames are optional. It does send the idea that it's just something you'd prefer to be called. But I think based on what you've said there is a bigger red flag going on with your sister and her husband. Your oldest nephew saying "Birth name, SHE" is concerning. That is not something that a 9 year old says on their own. He learned that from somewhere, so it seems likely that either your sister or her husband has made it a point to make this distinction in their own household.
I think what you should do is speak with your parents and anyone else that lives in your house (you didn't specifically mention it but it doesn't sound like your sister and her kids live there full time). Get everyone on the same page. No more "nickname" crap - HE changed HIS name and this is what it is now. I would explain to them that you're worried this will be confusing for the children if they're constantly getting mixed signals and information, and that you'd appreciate your folks being super focused on getting things right at least when the kids are visiting.
And while I do think you will need to have a conversation with your sister, I think it may actually be more effective if it comes from your parents. If your sister is anything like me, I perpetually see my younger siblings as children - half of them are adults and while their lives are their own, I do mentally nitpick every major decision they make and constantly question their maturity. If your sister views you similarly, that's probably compounded by the fact that she doesn't agree with your decision to transition. She may be more open minded to hearing from mom and dad that this isn't a decision you've made lightly, you've seen professionals, you know what you're doing, and this is the way things are now. I would also couple it with either a conversation or a note from you letting her know how much this situation with her has impacted you and how you'd like some resolution to it.
What also factors into this are your plans. Do you plan to stay at home indefinitely? Do you have plans to relocate in the future? What value do you place on your relationship with your sister, and has coming out created any signficant change in her treatment of you (pronouns and name aside)? Ultimately - is it worth rocking the boat? It probably is if you don't have plans to move, but if you're only going to be staying for another 6 months or a year and will be moving far away after that, maybe not. Has your sister typically been a big part of your life, or rather, if you were not living at home, would your family be something you prioritize? While it might make things comfortable in the here and now to get the kids and your sister on the same page as your parents, if you are not otherwise intending to be involved in family life if/when you move out, it may not be worth the emotional cost to your family now.
I rewrote this post a few times as I thought things over, and I do genuinely think the above (re: you talking to your parents, your parents talking to her) is the best course of action that I could come up with. Obviously you could take it upon yourself to explain it to the kids, but if your sister and her husband are as not nice as it seems, that would probably lead to a lot of drama. I would also worry that if things don't go your sister's way without her being convinced that it's the right move, that she would stop bringing her kids around and that would cause some internal drama with your parents missing their grandkids.
