I have the same hairline and it freaks me out too. You are not alone!
Some days it bothers me more than others, but let me tell you a secret:
KillBelle is right. Exercise and eating healthy makes it grow back.
When I was 24 my hair was thinning in places, I ride my bike nearly 40 miles a week and just eat small things throughout the day (about 1900 calories worth - doesn't matter what it is, carbs mainly), now I have a curl forest when it's dry and just the temple loss. I assume that AA's and HRT will make it even better. I still worry about it, but now I mainly fret about my face. I can't stand my face.