On the website it says:
"If your old licence has been lost, stolen or destroyed, declare this on the form."
So you might be able to get away with pretending that your previous license has been stolen, but they might charge you the £20 replacement fee for a stolen license then.
You still need the D1 form though, and my experience of ordering forms from the DVLA has been terrible, I think it took over two weeks just for the form to arrive, but this was over a year ago, so I can't really remember. I know that it was long enough for us to assume that it had been lost in the post and requested another one.
Taking your passport with you really isn't ideal, not least because it'll have your birthname in it, but it's just about the only other form of ID a lot of places will accept. Unless you came to my pub, we're a very trusting lot out here (and know 99% of the people).