I say... dress however the heck you want, lol! Who said you had to own any casual wear anyways? I give mad props to the dapper dandies whose wardrobes consist of nothing but brightly colored suits, pocket squares, bowties, cufflinks and matching socks, and the gals (and guys!) who dress up in their sweet lolita best every day of the week. I'd go out trussed up in a complete suit of foam armor all the time if I could, hehe.
And yay! Haha, yeah, INFJs seem to be a bit on the rare side.
You're right, it does feel wrong. It's something that I can mostly work with (at least right now, I can't say how I'll feel next year or in 5 years from now), and if it's more than just irksome for you then I sympathize. Not "passing" as cis is one thing, but definitely don't let other people tell you how you can and can't look so long as it's not something you're doing to appease somebody else. Yeah, I sound like a big hypocrite right now, but the least you can do is to arm yourself with impeccable self-awareness and know why you're doing the thing that you don't really want to do so you can formulate an exit strategy and cope as best you can in the meantime.