I get how you feel. I also feel a lot of conflict from my past experiences, but I'm trying to stop myself from using the past as a way to stir up doubt and invalidate how I feel now in the present. I doubt my "trans status" all the time, there are times I'm so scared I'm wrong in everything, because as a kid - sure I wasn't a stereotypical girl, but nor was I a stereotypical boy. I had almost no signs of being trans as a kid, and I remember wearing dresses as well as "boy clothes," liking TV shows/games that were more stereotypically "girly," but also those that were more stereotypically boyish. However, all of those were simply likes, hobbies, things I did that fit or didn't fit into gender roles - they don't define my gender in itself, and they certainly don't define who I am now.
For a long time I didn't question being a "girl" because it was what I was assigned and I went with it, and not until much later on in my life (more recently) did I begin to figure some things out about my gender. The past always creeps up on me though, with many conflicting doubts, but I have to remember that a) sometimes we don't remember things very well or very accurately, and what we might see as significant doubts might not have been how we remember them, and b) tv shows and hobbies and toys don't necessarily have a gender. We like the things we like, as whoever we are. Also - there may have been many factors in your childhood that influenced how you as a kid viewed your gender. As a kid, I may not have consciously been forced to be a stereotypical girl all the time, but undoubtedly even living in society, there was a pressure to conform to the gender identity I was assigned at birth. Many different factors go into it, and I have to remember to be more lenient and kinder with my past self when I reflect back on my childhood, and when I wonder why I had no clue I was trans for such a long time.
What's important is how I can move on more positively and more in tune with myself from here on out (it's a constant journey/self discovery); my past may help or give me some perspective, but at the same time - it was my past. I can't go back and re-experience things, nor can I ever gain full understanding of why I did some things/acted some ways, and not others. My past doesn't invalidate who I am now. Although they may cause me doubts, I won't let some of my "girlish" interests as a kid pull me back from being a guy, and transitioning if I believe that's the right way for me. Although I often feel like I have something to prove to everyone else, and to myself - especially since I'm quite a gender non conforming/feminine trans boy - I'm beginning to learn that I really don't owe anyone anything, nor must I have an explanation + justification of everything in my past to be valid in my trans identity.