I have a feeling this also would vary widely by age group. I see a lot of younger straight and bisexual men that would say they would date a trans woman for instance, than in the past.
I think if you do categorically reject one group of people, then yes you are being prejudicial.