The age at which you knew isn't important. True, the "classic" transsexual will have known, or at least sort-of known, since their childhood. But that's just the common, "most normal" story. Note the quotes. For example... it's as if I told you that the classic depressive person will have trouble sleeping. If you sleep fine but have all of the other symptoms, it doesn't make you less depressive.
Me for example. I did feel like I was a girl when I was quite young, but I didn't do much about it, and honestly most of the time it didn't bother me a lot and I forgot about it for long whiles. That's because in children, gender generally matters much less. It starts to make a bigger difference when you grow up and children around you come to really understand what it means to be a boy or a girl.
Also, it appearing around puberty isn't abnormal. Rather, it makes perfect sense. Having a female brain in a male body will feel much more uncomfortable once more testosterone starts running through it. If you're like me, it just makes you feel awful and not yourself to have the wrong hormone affect your brain. Not to mention that it's at puberty that bodies start to really differentiate.
There are extremes on both sides (and sadly I've personally looked like a "little man" pretty much forever), but some little boys will look really androgynous, and would just need to change to be completely taken as girls. If their genitals aren't a major problem (and to be honest to most young children, even trans, they aren't a major problem) and they're not too uncomfortable with the clothes, how would they know? But then, as the little boy begins to grow body hair everywhere, and his voice starts to sound weird, and his face gets ugly, and he grows way too tall, or whatever... Then what it means to be a guy is thrown onto his face violently. And if anything should feel wrong, well it's at puberty that it's going to hurt the most, whether or not it hurt before.
Not to mention that there are levels of dysphoria, and domains, too. Some people are actually mostly fine with living as their birth gender, but they CANNOT stand their genitals, and even avoid showering so that they don't need to see them. Others might be rather fine with their genitals and not see the point in investing so much money and effort into SRS, but will be extremely dysphoric when even their genetic-gendered name is called. And those are just two examples.
And that's assuming that you're being completely honest to yourself and haven't been manipulated. Some people should have known all their life, but lied to themselves until their, what, fourties, because to them or their loved ones, they didn't feel being transsexual was okay. I've personally been pretty sure I should've been a girl since I was maybe 10 or 11, after having thought about it for my whole childhood, but even though a pedopsychiatrist asked me about it specifically, I'd so many stereotypes and wrong ideas about transsexualism put into my head by my father that I denied it. It's silly to say, but until I reached the age of 18 and finally decided to request transition, I truly thought I was some sort of special magical being who would one day be able to reincarnate as a girl through some ritual or spell. If I was able to put that nonsense through my head for so long, imagine what people can do with more realistic things.
Being transsexual just means that you are not comfortable with your birth gender and want to live as the other, and will be happier that way. Hating your parts isn't a requirement. Heck, if you're okay with them, all the better for you. Surgery is probably not exactly easy to go through, and if you don't live in the right country, it's really expensive.
The core of the matter is whether, and in what ways, you are uncomfortable living as male, and whether, and in what ways, you would be happier living as female.
I don't have the answer for you, and maybe you don't even have it either. And it's okay if you don't know. There are professionals just for that, who can help you dig into your brain and answer that kind of question, so that you can decide whether (and if so, how) to transition and head down whatever path you choose confidently, without regrets. I actually highly advise anyone has gender concerns to see a therapist, ideally one who's especially knowledgeable about gender issues. It can't hurt, and it's likely to help you in lots of ways, the most important of which being that you can finally stop wondering and going back and forth, finally decide and finally move on.
Maybe you'll be able to figure it out on your own, and that's fine. But you could also remain unsure, and take a decision while deep down you feel that maybe it's wrong.
For example, you could take the path (and sadly it has happened to many) of simply going with what seems to be the simplest option because you can't decide, and bitterly regretting it later.
No matter what you do, though, don't base yourself on facts to take a decision. Things like "I never liked Barbies, so I cannot be transgender" (okay this is simplistic but you get what I mean) can easily add up and make a rather convincing fake argument "proving" that you are indeed a guy. And some family members and friends, because they are often very prone to denial, can very well "help" you paint that fake picture.
Thing is, this isn't a question of facts, but feelings. You'd be surprised at how many trans women there are who practically do everything like a stereotypical male, like drinking beer in front of football after coming back home from their heavy duty mechanic job, but are no less female. And I heard of several trans guys who still love girl-talk and make-up and nail polish, one of them actually enjoying crossdressing as a girl for some costume events, but are still very much guys inside.
A good therapist won't be bull->-bleeped-<-ted by your convincing lies, be they made up by you or others, and should help you make sure you find the true answer within yourself. You may not always agree with them, but their actions should (normally) always help you look in the right direction and come up with your own answer that feels right.
Anyhow, whether you do it alone or not, the important thing in your decision processis that it must feel right with you, no matter what anyone tells you, and no matter what's easier or simpler in practical terms.
PS: Oh yeah, and in theory, crossdressers are part of the wide "transgender" term. It basically includes anyone who doesn't feel completely comfortable as their birth gender, I think.
Edit: Forgot to tell. How I knew for sure I was transsexual and decided to transition was when I thought to myself... "I want to be a girl and nothing about being a guy feels right to me. Sounds like the definition of transsexualism." Mainly what lead me to this was thinking about how whoever I envied was either a girl who looked happy and cute, either a guy who had a girly (thin, with a waist, less body hair, cuter face, etc.) body compared to mine. And how the fictional characters I've identified the most to were overwhelmingly girls.
Then I listened to a song I like (Q by AAA if you're curious) on my iPod while thinking about stuff - I was hospitalized and bored, which was a pretty good environment for thinking - and remembered the music video for that song, in a part where there's a close-up on the lead female singer as she sings the refrains, looking cute and especially happy, and I thought about just how much I'd like to be her.
There were other signs about my personality, tastes, dreams, feelings, etc. but that was the triggering event, and when it all came together and made perfect sense, as I ditched my occult explanations.
