To put things simple when the testosterone in your body goes away, that is what fuels your stamina, alot of energy, strength. Without it you will tire quicker, have less energy, and be weaker. It will inevitably affect your martial arts. Why do you think as a general whole women aren't as strong as men, and they have to work harder at physical tasks? Its not impossible, but it does make it more difficult .
If this is a concern, understand without testosterone fueling your body you're going to have to rethink some things. I still do martial arts, but I also tire quickly and my muscles are sore as well.