I'm very sorry for how that's happened for you. However, I think you can patch things up.
I think you were right- being direct would have been best, kids are usually good at understanding something if adults tell them directly.
Exactly how to patch him up, I'm not sure... I think your first step is to make him feel secure knowing that you love him and aren't going anywhere.
After that... I'm not sure exactly how to tel la young child. What did you have in mind?