Have you heard of the famous actor Rowan Atkinson? Or Rowan Williams, the recently retired Archbishop of Canterbury?
Names that are clearly female typically end in an audible vowel, so you're lucky that your name doesn't follow that pattern. That's why it's considered a unisex name.
I think that the answer to whether you should change your name or not comes down to whether you can tolerate living with your current name for the rest of your life. Many TSs change their names as way of breaking away from their old identities. Speaking for myself, I am content with not changing my name. But then again, my name has always sounded female, so maybe it's not a fair comparison, LOL.