We can play with semantics and morphology all we want, but the bottom line is that -phobia has a clear and consistent meaning in our society. And it has utility.
As long as we live in a society that considers fear shameful, calling the haters fearful is to our advantage politically. Add in that the "fear" is nonsensical, and as public awareness grows, the haters appear ever-increasingly absurd.