They're no doubt in a bind - the Master/Mistress is too classic to just retire forever. It's a bit like with superhero comics, the Joker has become a bat$#&% boring character but every writer wants a crack at writing him despite generally diminishing returns. The general public expect to see him too.
To their credit Doctor Who came up with some new ideas and concepts as well. Some work, some don't. But the Master, daleks, Davros, cybermen and now the Weeping Angels are all too well known to be written out. I think they use them for the big events because the general audience (not just hard core fans) understand immediately what is at stake.
I really enjoy Capaldi as the Doctor, I was skeptical when he was first announced but he has quickly become my favorite Doctor in a long time.