I feel there's a lot of great replies while I have been writing a book here, ha ha... I agree a therapist qualified to discuss your feelings would be the best place to start! Only you can say what is going on and a good therapist is there to help you find out what will be best for you on your way to feeling more comfortable with your expression of self. It is certainly helpful to talk these things out online or with a Transgender group if you can find a local club as well

Having friends and support that can relate to what you are going through is exceedingly helpful.
some trigger warning for discussion of fetish, perversion,prejudices and consent issues:
I just finished reading the book "Pervert: The Sexual Deviant in All of Us," by Jesse Bering. He's a gay author and has written other books on sexuality that I haven't read but I really enjoyed this one. He discusses many subjects and as a warning he does discuss fetishes like scat and pedophilia, but as a way of understanding sexuality as a whole. Basically, as I see it, we form our basic sexual drives and attractions (fetishes, not attractions as far as sexual preferences like being gay,bi or straight) at very young ages. It's a great book for anyone to read in my opinion and it has many references to other more science heavy books that could be interesting to follow up on! Please be aware that I use the word pervert to refer to myself as well as others and not because I am transgender. To me, it means 'not average' and I generally consider it a good thing or a very neutral label. I believe the majority of humans (and even many animals) engage in or imagine sexual acts that do not involve reproduction, and many of those behaviors are consider perversions but are not at all bad or unpleasant, just different expressions. Most are relatively harmless but many can become harmful if things like consent and communication are not involved, so I'm going to do my best to explain my views...
(I am also going to toss in a quick disclaimer: have you read about genderqueer definitions and concerns? You may also consider yourself genderqueer or fluid if some days you simply feel more feminine and that is a perfectly valid gender expression!)
There's a section of the book where he discusses transgender fetishism, which he makes pretty clear is tied to but also different than being transgender! It's COMPLICATED, as there aren't necessarily distinct terminologies to describe each way distinctly. It really sounds like you may be a fetishistic transgender person, rather than a ->-bleeped-<-. A ->-bleeped-<- (as I see it) is happy as their born gender/sex but enjoys cross dressing either as a daily comfort, or as a sexual fetish. These can overlap, but usually someone can tell you if they like to cross-dress as a sexual thing, or as just who they are on a day to day basis of feeling more comfortable in a specific kind of clothes/appearance. So like being a cross dresser, one can be transgender as a day to day "this is who I am" thing, or more as a fetish. I don't feel this diminishes your feelings in the slightest, but I would caution being very sure of what you want if you plan to take hormones or get any kind of surgical work done.
That being said, your average transperson is unique! Some people are perfectly content forgoing surgery or even living as their birth sex- looking, dressing and acting the part of the gender they were assigned at birth. In my experience, the vast majority of transgender people need to do something to alleviate their disphoria, but everyone's remedy of choice is different. You should not have to feel any different there- your journey is your own. But since you are not constantly disphoric and as I see it, your disphoria appears to be more linked to your sexuality, I would caution leaping into a permanent change until you are sure it is right for you! Don't feel you have to doubt your feelings, explore your options and think about all the pros and cons of each.
Do you feel like having breasts would be an important thing to you? Would you be happy with forms, or do you really want your own body to look more feminine every day? There are many excellent products that can change your appearance and MANY people that would be happy to accommodate your desires if you have good communication and honesty about yourselves. Know that your situation, your feelings might change and that is okay! I just wouldn't wish for you to 'go too far' so to speak and regret changing yourself in a way that would be difficult to come to terms with. Though this is true for anyone thinking about transitioning issues. I assumed I would have to want bottom surgery, but many people are happy forgoing an expensive, extensive operation and I am one of them. I am pleased that there are people out there who are tough enough to go for it because it is what they need, but I really do not expect anyone to charge into any kind of operation without considering the graver consequences and be willing to accept the possibility of complications that are inherent with all surgeries. Same with hormones- some people take them for a limited time, others will want to remain on them for life! Everyone is different and there's never really a one size fits all solution for humanity. Plenty of lovely ladies who are happy with theirpenis and many guys who enjoy having a vagina. I for one am going to remove my ovaries but leave everything else intact. I am having top surgery and care very little about my nipple sensitivity but I am probably never going to be okay with the idea of losing sensitivity in my genitals. It's just a risk that is not worth it to me, but if someone else is needing to change their genitals I am happy to support them and wish them all the best!

I would recommend writing down your feelings, even just for yourself, being mindful of how you react emotionally, mentally and physically to women and the idea of being one. It seems you've already been pretty honest with yourself, really the important thing to do is figure out how and when you want to express it! A therapist could be helpful as well, you shouldn't feel guilty for having drives and feelings that differ from other transgender people , and I feel like you can still identify as transgender if that is how you feel, or if you need to have a new label, you can say that you relate to transgender but are not a full time transgender person. Not everyone will understand, and you may want to moderate some behaviors or not act transgender all the time if that is how you feel. I guess the biggest concern here is if you mostly identify as male or masculine, you will need to respect women's spaces as best as possible. If using the women's room is something that arouses you, or doesn't feel like a necessity (rather than makes you feel safe and normal) I would really say it's best to use the men's room, or neutral bathrooms rather than the women's. Be aware that men have a lot of assumed rights and priviledges and you don't want to be invading a place that is considered safe for women, trans or cis! There is certainly nothing wrong with you, but it seems you might have some fine lines to be aware of and responsible to not cross as a respect for others. There's nothing wrong with having an unusual (or not so unusual) fetish, many people are willing to indulge their partners in turn for their partner indulging them, or seek out others to fulfill their own needs. There's women out there who love men who crossdress and would be into you being a woman for sex, or when you feel like it. The only time I disagree with acting out one's needs is when they are hurting or imposing on another creature. It's not okay for a pedophile to act sexually upon a child, or a zoophile to accost an animal, but roleplay, imagination and consent are things that make possibly difficult fetishes bearable and healthy. It would be perfectly okay for you to go out with a person while dressed as a woman, but you will have to make your own rules pretty clear. If you are dressed up as a woman, does that mean you are a woman for the night? Does that mean you feel safer using the ladies room? Be aware that there can be consequences and dangers and please do you best to to use other people as part of your scene if in fact you decide your transgender feelings are sexual in nature. Its is rude to humiliate or impersonate for your own gain, be kind and respectful to others. I don't want to imply you need to hide your sexuality or gender expression, but think of it this way: Unless a BDSM person is at an event where it is allowed to strip and beat their partner, they should not do things that are BDSM in public places. It is expressly illegal to engage in sexual acts in public, even though some people have exhibitionist needs. A flasher is a criminal, a dancer in a topless joint in not! Context is very important and it would be rude as well as criminal to engage in inappropriate behavior just to get yourself off for the day. I even disagree with a couple in my area who engaged in light behavior in public that was bringing the fetish to innocent bystanders. I've regretted doing similar misbehaviors myself because even if it isn't illegal to drag someone around on a leash in public, it can be rude to those who have not consented to be a part of that scene. To me, it's important to make a distinction between the everyday and the fetishistic. If someone is mostly aroused by dressing in women's clothes, please consider that they are, in a way, expecting strangers to be a part of your sexual experience. That being said, I don't disagree with someone wearing lingerie under their work clothes, or even wearing a subtle packer or using a walk around toy in public, provided that NO ONE EVER NOTICES. Humans seek out thrills and novelty, but there's a difference between naughty little secrets and flashing strangers in an alley! There is also a massive difference between women who dress and act like women as part of their gender identity and those who are more a part of a man who needs to indulge in their feminine side for various reasons. I will still happily consider you a sister/brother/other transgender no matter what you decide. In the end we are all human and need to work together regardless of our genders...
Some words of caution: In my opinion, it is misleading to say you are fully transgender if in your heart and mind you know better. If that's the case, you would need to be careful about when and where you perform sexual activities and do everything you can to respect innocent bystanders as well as be an upstanding person because if you are outed, most cis people will lump you as a transgender person and assume your behavior is indicative of all transwomen. Misbehavior by few tends to make the conservative majority leery of 'perverts', even though very few humans are NOT perverts(in my opinion and research, ha ha!) Sad to say, you can still be a hate crime victim at any time even if you are only partially transgender. I make no real judgements as to what is your unique expression but I know things like fetishistic ->-bleeped-<- exist and have a lot of unfortunate misconceptions that harm both the transgender and the fetish communities. I feel like a lot of transgender people are much more aroused by the idea of being their true gender, so it's not that that is unusual in itself, I just don't see much serious discussion of fetishes in daily conversation beyond "those people are messed up" judgements, except among forums like this as well as others that I visit that are more fetish specific.
So these are my long rambling thoughts and I hope any of this helps. If you have any need for private PMs I would be happy to discuss privately or further here in the forums