The only time I have issues with altitude is when I do a quick ascent, like driving to the top of the Continental Divide. If I don't exert myself, I am ok. But hiking or prospecting above timberline gets me winded pretty quickly.
I am at about 4,000 feet elevation now, and my mining site is at just over 5,000. There isn't much difference, so it doesn't bother me. I have hiked in Rocky Mountain National Park, climbed Pikes Peak twice, and Longs Peak once. You just have to take your time and drink plenty of water.