Sounds like your ex is very unlikely to launch a compelling legal case. It's good to be prepared and having a qualified family-law attorney's number handy is a good idea, but for a fraction of the cost of retaining an attorney I'd get a few security cameras with off-site recording and aim them in locations where you'd like to know everything is okay. Pay attention to monthly fees and how long you have access to recordings, that's more important than hardware cost. Some people find security cameras annoying but the fact is if someone isn't doing anything wrong it shouldn't bother them.
Replace your front door lock with an electronic lock - a keypad, or one that uses a smartphone for access. At that point you can change all the physical-key locks or simply ask for your old key back. With some electronic locks you can get a time log showing when each authorized person used it, restrict access codes to specific hours of the day, and cancel an individual code if necessary.
