I think most people will find young (~18-25~) people at least as attractive as older people, and more often than not, more attractive. People are made like that: their body hasn't started to "decompose" due to age in those age ranges, so obviously, they're supposed to look better.
The difference is that most older people would not appreciate their partners' youth for its moral aspects - irressponsibility, impulsivity, high sex drive, etc. Or fear the social stigma. But I don't think the physical attraction to a young, fit body is frequently absent.
Don't be ashamed. As long as it's legal, there's nothing wrong.