I, personally would not be in female only spaces. This is why I avoid women's bathrooms, and feel kinda awkward at baby showers :/
I don't fit there. I would be making myself uncomfortable, as well as others.
I suppose, no, I cannot tell another guy that he is not allowed to go there, or should feel ashamed for doing so... but most transmen are trying to be accepted and treated as guys. Not only is this seeming to be taking a step backwards in transition and invalidating yourself, but if you are trying to be accepted as male, society will not usually allow such coin flippage at your leisure. Most of the time, its just a "pick one" deal. I understand that this is painful, but we have not changed everyone's attitudes towards gender/sex and we may never.