I own a pair of converse chucks. They are easily available, comfortable, reasonably durable, lightweight enough to get good clutch feel, and come in small sizes.
While they are still expensive, they aren't as outright insane as skate shoes.
You could always try those. Mine have been my only shoes for over a year, maybe two, and they don't show signs of breaking any time soon. Also, unlike other fashion shoes they're said to look better the more engine oil and rust converter you get on them

And for the record, mine are a men's 4 (though they are getting quite snug now. I will have to go larger soon).