I've actually asked several healthcare professionals about this, since I'm coming up on the age they used to recommend getting a baseline mammogram. The consensus seems to be more or less in line with my own assumption - our overall risk is higher than a cis guy's (which is, as people have said, not zero), but lower than a cis woman's of the same age. That's disregarding familial and environmental factors for a given trans woman.
Some breast cancers *are* encouraged by estrogen bathing the tissues, so presumably at a minimum the risk is equivalent to the breasts' age if not the woman's (=on HRT for 10 years, roughly the same as a 22-year-old cis woman, etc.).