Two things to try:
Enjoy your visits, as new experiences. Treat the visit for the most part as Ashley's first trip to see these places, after you have heard so much about them from "him". (I've been re-visiting places that I enjoyed growing up, and I have had a lot of fun approaching them a bit differently, as myself. 'His' memories are there, and really my memories, of course. I treat them as a sort of old travelogue, and like to note what has changed.)
If you do feel that conflict coming on, try a little mindful meditation to let go of the conflict and distress.
The hardest part may be encountering your old friends. Don't surprise them with yourself, and I suggest avoiding contact with anyone who is not accepting.