Yes, specifically with children.
I'll elaborate; people have no problems with me being around children as a woman, but as a white dude, I was automatically under suspicion of being some kind of creepy pedo guy if I showed any interest in kids.
I will comment on the robbery target thing though - while I was probably more likely to be knocked over for my wallet as a white dude, my chances of being raped were virtually nil. Some days I'd like those odds back

"Y'all don't know what it's like - being male, middle-class and white."