A hate crime targets a group, not an individual. That's the key. There is the crime against the individual, which is significant. Someone dead, for instance, is dead regardless of their skin color, sex, gender expression, sexual preference, religion, etc. And it's just as significant no matter who they are. But many crimes stop there - they were an attack against an individual. But other crimes are an attack against a community of people. Sure, only one person was attacked, but the attack was intended to send a message to the entire group: you aren't welcome, and I'll (insert crime) if I see you.
If I am walking down the street and someone attacks me, maybe beats me up, but didn't do so because of bigotry against some group I'm part of, it might concern people who frequent that area. But most people probably wouldn't be particularly afraid for themselves, at least not any more or less afraid than they were of that area before.
But if I was attacked because I was a member of a certain religious group while walking to my church, that gives the entire religious group a reason to feel afraid. It was an attack against them, not just against me, and certainly not random.
Yes, it's hard to prove intent. (the following is likely US-centric, as I don't know much about other legal systems or cultures) But intent is used in many prosecutions - for murder vs. manslaughter, for instance (and the degrees of murder as well). It is definitely used in determining sentencing - did the person intend to do great evil to someone else, or did they do it because they felt trapped and had little choice? Both may be crimes, but one may be a significantly longer sentence, due to intent. This works in civil cases too. Intent is already part of law. (a scary trend is the new trend of creating laws that don't require criminal intent for a person to be guilty, but that's for another post). The Latin term for this is mens rea and is a relatively old part of the legal heritage of the US and at least a few other nations.
The second reason for hate crime laws is that often relatively minor acts of violence (the hate crime laws I have seen have a relatively high standard for proving guilt, and require physical harm typically, not just thoughts of hate) would be seen as less than they are. If I was a Jewish person, and someone beat me up because of it and carved a swastika into my arm, that should be treated as a different crime than someone getting into a bar fight.
The third reason for hate crime laws is to guide juries, particularly when deciding things like whether "trans rage" is a legitimate defense. Without the hate crime charge, juries will use their own prejudice in deciding what is reasonable - and often sentence the person to a lower charge. The hate crime laws help treat that type of criminal act the same as other similar criminal acts, when it would otherwise be minimized and treated as less criminal. If this bigotry didn't exist, then this would not be necessary. But it does exist. And by having the charge at a trial, the jury knows not only does murder/rape/assault/battery need to be considered, but it's even a worse crime if you did it to target an entire class of people - it's not a more understandable crime, it's a more significant crime.
As for blacks being prosecuted for hate crimes, yes, that does happen. People like myself (a white person) don't need to be screaming that we're being unfairly discriminated against. We're not. And just like every trans person being attacked by a non-trans person isn't necessarily a result of hate towards trans people, not every white attacked by a black is a result of racism. But it is a lot more common for a trans person or a black person to be attacked because they are trans or black than it is for a white person or a non-trans person to be attacked because they are white or non-trans.
So, yes, crime might all be hate, either specific to an individual or directed to a specific group. But there is a distinction between crimes with one victim and crimes with a community as a victim. And it is reasonable for the law to reflect that distinction.
