I'm not even sure that any part of that is a bad stereotypes, (hell, it's a major tourist draw) it's just the wrong neighborhood, and where Colfax is bad, and has been bad, the Castro is nice. You should be so lucky as to live there. One of the warmest parts of SF, 10 minutes on a bike to downtown, awesome theater, killer views, best places to eat - what more do you want out of life?
(I think I was introduced to Colfax in HS because Kerouac talks about it in On The Road. The Cassidy character - Dean - lived out that way, it was ->-bleeped-<-ty then, and that was BEFORE the interstate really drug it down.)