The patches that do grow well, grow them out really long and then braid them.
And then put little beads on them.
And then call yourself Jack Sparrow.
Uh, er, I mean, lots of times there will be small bald patches that never go away for years or at all. If you think you can make it look good or make it work, I say keep it.