Dude, this idea that liking horses is feminine is bunk. I love horses, and recently I even took more of an interest in horses like Friesians and Gypsy Vanners, and I have it from other people that the most feminine thing about me is my recent worries about weight (which guys have) and my interest in clothes and looking good (which guys do). Think of it this way: cowboys, cavalry, historical generals and emperors -- all rode horses, all were men. Think of all the historical pictures of powerful men on horses and tell me that's feminine.
Honestly, owning a Pomeranian is more feminine than liking horses, and plenty of men own and baby small dogs without drawing any question to their masculinity. I've even seen a man who in no way was not masculine picking out a pink coat for his dog -- he appeared more involved in the process than his wife.
Sure, horses are a more expensive animal to keep than the average dog or cat, but it's not as though only the rich own them or they cost millions or anything. Leasing is a good way to start and gives you a good idea of just what kind of care is involved -- volunteering at a barn gives you a pretty good idea as well if you want to give that a shot without the commitment or the money if that is a problem at the moment. Otherwise the best advice is to be prepared as possible and do as much research as you can beforehand. Understand the basics, then research the beyond of keeping a horse, because really it's the beyond that can prove the most beneficial and can make all the difference.
Also, know that a horse is not a horse is not a horse. Learn about the different horses and what makes a horse a good horse opposed to anything else. Honestly, knowing as much as you can beforehand can only benefit you.