For me it was a combination of being a social outcast and finding that computers were an interesting puzzle to figure out. The solitary work conditions were an added plus. I have personally seen a lot of strange personalities in this industry - people with horrible social skills, or conditions that could be described as "asperger-ish". Most of these "weird" people are truly brilliant in some way. When it's just you working with computers "weird" doesn't matter - what you can accomplish is the only thing that matters. Computers don't judge you, and weirdness can be an asset.
And speaking of math - I am not a math whiz by any means (I can add, subtract, multiply, and divide LOL....), but I find that there is plenty of work in the industry for the non math gifted people too - people that are good with logic skills, people that are good with communicating with both the business people and the programmers (translating between them), or "big picture" people that can understand and solve huge business problems.