Well, I'll tell ya, my girlfriend, who is normally a small or xtra-small is always raiding my closet filled with larges, because sometimes, she just wants a loose fitting item. So it depends on how you feel and the look you are going for.
On you though, the medium, just shows off your straight, male body more.
You might even want to try xtra-large.
And remember, they're probably going to shrink in the wash.