I'm in the same boat. And sometimes that makes me think I'm just making it up. But just because I didn't know until puberty hit me, doesn't mean I'm not trans. When I was little, I didn't really care if I wore dresses or not, like I wouldn't wear one on purpose, but if you put one on me, I wouldn't object. I was a bit of a tomboy, since I didn't really want to wear makeup and stuff, and I didn't like playing house, but I really didn't know I hated being a girl until much later. Now those years beforehand haunt my memories, but if I ever doubt myself, I just remember that I've felt this way for quite awhile now, and if it was a phase I would have gotten over it a long time ago.
Anyways, long story short, it doesn't make you any less trans or male than if you had known since you were 4.