Having grown up in a small town, and seeing the prejudice that can occur, I am happy to see that things appear to be working out well for you in your new job.
As for the passing, people will see what they want to see, and if you offer no alternative to being female, then people will look at you and address you as female.