I think regardless of muscle attachments, making your muscles bigger will make you look more manly. A lot of women get away with big muscles because their other features are extremely feminine, or at least there's "something" that makes them definitely un-misgender-able. But still, all of those would, at a 99% chance, look more womanly with less muscle.
You can be in shape and not have very big muscles, though. Depending on the effects of your HRT and your genes, maybe you can even do weight lifting and end up with super small, yet "solid" arms... But that's mostly if you're a very, very slim person to begin with.
In the end, it's still your choice. But keep in mind that with all the "little male things" most of us retain, it's quite risky to do muscular training. If you do, though, I'd suggest avoiding very masculine-looking muscles when they get large, such as calves, arms and those side-of-the-neck muscles I don't know the English name for.
