There is no "supposed" here. You can shop wherever you want. No one will give a second thought if you shop for men's clothes, either in a men's wear store or a department store. Half the men in North America probably couldn't buy themselves a shirt if they had to. Their wives buy their clothes.
However, you may still find it easier to find androgynous clothes in women's stores. Society considers it fine for women to appear masculine, but totally unacceptable for men to appear feminine. So men's clothes will mostly be unmistakably masculine, whereas some women's clothes can be in the middle.
When I was shopping for androgynous clothes, I went to women's stores or departments. Plain women's slacks, a women's top that isn't particularly colourful or frilly, and sneakers aren't going to scream feminine. If you want to present more masculine than that, try jeans and a T-shirt, the uniform of half the male population. They can be women's cut, to avoid looking all the way masculine, while still being something that a male would wear.