I think that mixed signals are part of the non binary or andro experience. HRT does change how you think and feel and depending upon a number of factors (dose, genetics, age etc) it will impact body shape etc. Even when on low dose my body reacted and changed. While physical change was not as important to me as addressing my dysphoria, it happens. You can't pick and choose.
However you do have a number of tools available to you to either amplify or to tone down the physical changes. Personally I am enjoying any change which takes me away from the alpha male physique and presentation that I was burdened with for so many years. Yes I have had FFS, I have permanently removed all facial and body hair, I have grown my hair to shoulder length and my brows and nails are manicured and even with this I am usually gendered as male or perhaps andro male, if I am in a work situation (due no doubt to the 'male' uniform, physical size and voice). If my jacket stays on or if I have a loose shirt then my breast development is a non issue.
Now once I tell folk that I am trans, non binary they usually gender me correctly and better perceive who I am. However I suspect that ordinary folk really don't pay a lot of attention. If I wish to 'play' with gender then I subtly change my presentation - skin toner, mascara, a natural lip color, tie my hair differently etc. This helps me be perceived as andro or andro feminine, if I marry this with more andro clothing.
Really the only time I am self conscious is if I am swimming/on the beach; I wear a tight t-shirt; I am hiking (the chest straps make my boobage quite obvious); and/or am with folk that do not know me as non binary.
As Sandboxed says, gender confusion is not a bad thing. I find it validating.
Safe travels
Aisla