Large, large, large doses of phyto-estrogens can do something. Unless you take supplements or have soy milk plugged directly into your veins, you don't have doses large enough. Also, I'm not 100 % sure, but I think products like tofu don't have it anymore in them.
Anyway, you shouldn't worry about it at all. Whatever the effect of soy is, positive or negative, it's not significant enough in any normal case to care about.