I just posted earlier that I always felt "different" or less than "normal" on the inside but could never identify what that feeling was. Over the last few years, like you, I would wear my girlfriends underwear on occasion when alone( or at least tried, but they are way too small for me), which brought back memories of doing this with my mothers when I was home alone or up late at night as early as 8 or 9. Even then I had no idea what that was but I kept it to myself and by my teenage years had repressed the memories almost completely until about a year ago or less.
I also began looking at a lot more transexual pornography over the last few years and I eventually realized I was seeing myself as a transexual when doing this, and that it stemmed from something much more than just having a fetish.
The bottom line is that I think that everyone realizes it in their own time, and the lack of understanding early in life has a lot to do with societal pressure to "be what you're supposed to be" whether that be all man, or all woman. Either way, to answer your question, I can't speak for everyone, but I don't think even cis-gender people are 100% sure of who they are or what they want in life. From this perspective I think it is safe to say that it is definitely true that a good number of people have no idea they are transgender until they are truly ready to face it within themselves.
Sent from my iPhone using Tapatalk