I wouldn't say it's shallow, or at least not unreasonably so. Remember spacial that those of us who are pre-T or in the early months of T can look really, really young - I mean, I'm 27 and I look like a 14-year-old boy. And that's being generous. Maybe 13. It would be really unfair to mid-20s guys to blame them for not being attracted to me as a guy...actually, it might be a little disturbing if they were, at this point.
Tripp, it's not necessarily wrong to date younger girls, but (1) be aware of the laws in your state and respect them, and (2) be aware of a possible maturity difference that can make it hard to establish a relationship as equals. Any relationship between a 16-year-old and a 20-year-old has an inherent power imbalance (you have legal rights and freedoms as well as possibly financial resources she doesn't have, not to mention a bit of life experience).
Also, be prepared to deal with questions she might have - not just about you, but about herself and her own sexuality - when you disclose. Nobody can predict in advance how a particular person will react to disclosure, but you can expect some confusion, which can manifest itself as some very strong emotions in some people.