I designed, drew up the plans and built my house in 1986. It's 4BR, 4BA, 2500 Sq/Ft with a full basement. With help, it took me 8 months. I contracted out the concrete, framing, HVAC, roofing, siding, drywall taping and carpeting. I did the survey (for locating the house on the lot), electrical, plumbing, insulation, drywall, fireplaces, trim carpentry, tile work, painting & staining and some inside framing for the fireplaces.
I'd work 8 hours on my regular job (electrician) and then go to the house for another 6-8 hours, M-F. On weekends I'd put in 10-14 hours per day. When I started I weighed around 150. When I was done I weighed 128. Once we moved in I couldn't sit still for months. I was so used to long days a regular 8 hour day just wasn't enough. I was wired.
As far as physical strength, what you need all depends on how much you plan to take on yourself. I did some crazy things, like screwing up 4'x12' sheets of 1/2" drywall on the ceiling by myself. But if you use your head and contract out the work you can't do yourself, physical strength isn't all that important. Stamina is though. It's a lot of work!
If you act as general contractor and you don't know anyone in the trades, do your homework and get references for everyone you sub-contract to. Don't pay anyone in full until the job is done. Never pay more upfront than the cost of the materials on site. Watch for overages in materials by getting a detailed list of all materials shipped to your house. Some subs charge materials to one home and take them for personal use.
Get insurance! This is a must! Don't even think of building without it. Even if one person helps you out, it's worth the money. Accidents happen.
It's a major undertaking but it can be done as long as you plan well and don't overestimate your capabilities.