If you're small chested you really don't need a lot to bind, and from what you're describing, you don't have the right size at all. I'm not in any way small chested and I can do 3-4 hours of Taekwon-do going hard at it with enough compression to pass and it never, I repeat, NEVER hurts. It's not comfortable, not in the slightest, but it doesn't cause pain. When you talk about being able to take a deep breath without one now, that's just another red flag. I can take a deep breath with mine. That's why I can do Taekwon-do in mine. I can wear fitted clothes and pass so it's not that I'm going too loose, it's obviously enough to pass.
So to bind or not bind? I'd say not bind with what you have. But to deal with dysphoria, I'd suggest going a size up and if you're small, just get a half top version that's more like a binding bra. Then you can bind when you need to.