Have you been on just estradiol valerate for 3 years and no anti-androgens like spironolactone or androcur? If so, that might be your problem. There's only so much estrogen can do when you have a load of testosterone pumping around your body.
If I'm correct, you really shouldn't be scared of taking anti-androgens yes there can be risks just like with any medication, usually if you take an unrecommended dosage but they help the estrodial to feminise you so much easier. And they really aren't as dangerous as you think as long as your not stupid with them and ignore whomever is prescribing you.
* You have a GP/endo for a reason, to check your blood and make sure that the medication doesn't harm your liver.
* Unless you're eating absurd amounts over the recommended dose that no GP/endo would prescribe you should be pretty safe unless you have a damaged liver to start from say heavy drinking.
If you are on anti-androgens as well and are in the age range for breast growth then I dunno really, I wouldn't put it down to being skinny because I'm 20 years old, 48kg (104lbs) and 5'8" (175cm) giving me a BMI of 15.4 which is dangerously underweight and I've still got breast growth after 3 months and 18 days on HRT which started to develop after like 3 weeks in (albeit because I'm very skinny they are growing quite slowly, can't fill out a cup size but they're there).