First off, don't wear underwear 'for your doctor'. If they aren't what you want, go with boxers. Or nothing, if he's such a jerk about it.
If he asks why you are wearing boxers, just tell him that he is being unprofessional and you don't want him to continue being inappropriate with you. That'll shut him up, or open him up to a lawsuit, depending on his reaction.