I've dealt with this problem, too. I'm always afraid of not passing, and someone being able to detect that I'm trans and potentially ruin my day. It's not really all too irrational considering statistics, but I haven't run into any negative experiences so far. It's likely my anxiety saying otherwise.
May as well live life, just be wary of the more shady areas and remember that anyone could potentially be targeted. If we all knew we had the risk of being jumped, raped or killed, and we decided to live in perpetual fear, nobody would be out and about. Being afraid to experience life isn't living.