I had been clothes shopping at all kinds of stores for my disabled wife (when she didn't feel up to leaving the house) before I ever started buying women's clothing for myself. No one ever bothered me or even looked at me strange. We wore the same size, so if something was questionable I could try it on to see if it would fit her. Even when I started buying for myself, half of everything was for her usually.
I only ever got hassled once, by an idiot, and that only time was after I was presenting as female. It wasn't a thrift store.