Jay, I think it might be a generational divide. Has your endo been to men's sections of even department stores lately? It's
all more androgynous than men's fashion used to be, it isn't just the more boutique-ish stores like Hollister, Abercrombie, Pacsun, ect. Even Kohls and JCPenny's are full of bright colored, close-fitting plaid button-ups, slim fit or skinny jeans, and hoodies with ornate designs and strange colors and finishes. Purple, teal, baby blue, pinkish red, neon green: these are the colors I keep seeing in young men's fashion, and it's not just for the gay boys either, or even skate/scene kids, most young men wear them. It sounds like what you were wearing was probably far from the most 'unisex' looking you could have selected from said men's departments.
Like Caseyy said, with things like shirts and jeans, you'd be hard pressed to tell whether they came from the men's or women's sections sometimes.
Anyways, you're 20, what the hell else should you wear? Also worth noting is that it's really hard for a lot of transguys to find sizes that work for them, I'm damn glad for newer, tighter fitting trends. I resent the notion that because we're in transition we should try extra hard to select the most masculine clothing we can with no regard to whether we like it or it's flattering or not.
Trust me, this guy's just out of touch. You know what I wore to my first appointment with my GP who prescribes me T? Slim fit, indigo wash jeans and a v-neck black t-shirt under a navy blue and dark grey plaid Hurley button-up shirt; the longer, fitted kind. I think that's more 'metrosexual' than what you were wearing, and my doctor asked me 'so, how long have you been dressing male?', when asking me about time lines with binding, coming out and such. It's really such a subjective thing, but I'm sure it's obnoxious and tempting to 'educate him'. What does he wear anyways? Loafers, high-waisted tapered-leg khaki's, and ugly golf shirts? Seriously, don't sweat it, he just doesn't appreciate your sense of style.