firstly, gigantic "you rock"s for the watchmen avatar.
secondly, while I am one of those who are attracted to a person for 100% non physical reasons, I can understand how if you're not attracted, you're not attracted.
Sex, or rather, physical intimacy is one of the key things in a fully functional romantic relationship. If you aren't able to feel the desire to be physically intimate, it is the only right thing to break off the relationship. Doesn't make you an a-hole, makes you an honest person.