Delay, stall, string him along, be coy. Just because you promised doesn't mean you will. It certainly doesn't mean you have to -- get that notion out of your head entirely. If a guy manipulates you into promising something you're not comfortable with (regardless whether he was even aware he was doing it), don't ever feel obligated to keep that promise.
Once you realize you don't have to go, you can decide whether you want to go. And maybe you do. Kayaking is fun.