I go through this inner struggle myself. I REALLY want to wear military uniforms, but people would know by my gut that I'd likely had no experience in the military. I am inclined to agree with the others though that as long as you avoid the patches and awards, etc., then you should be good. I think the U.S. Army is about to switch over their uniforms soon, so there will be even more tons of surplus. Besides, a lot of militias, wear military uniforms.
There's one guy in this documentary who is wearing a military uniform, and he has to be 300 + pounds (4:40 - 7:12). If he could wear one, I'm sure you could! :p