I hate it when that happens. I never perfected a polite way around it, though I certainly expressed my dissatisfaction whenever my parents gave me compliments like that, and they never caught on. Stuff like "yeah... thanks... how about you compliment me on my broken nose or something that won't make me feel like ->-bleeped-<- next time? I don't want to be 'pretty,' I don't want to look like a girl, and I don't want compliments that make me feel like one." From the time I was 9 until I came out when I was 15. How they never got the point... I will never know.
With my grandmother I was more joking, since she's such a pain in the ass that being that confrontational would make it worse. I'd say something like, "Really? That's a shame. Means there will be more boys to beat up when they hit on me." Actually, true to this, I scared everyone away who might possibly hit on me because I projected an attitude of "call me pretty and die." Haha, poor dudes.
On the other hand, in situations with people I don't know well, I'll say something like "Eh... I dunno, I kind of like the androgynous look. But thanks." It makes me look a bit alternative/butch, but androgyny is being increasingly accepted in the mainstream (even in fashion - tuxedo jackets, ties, structured/figureless clothes for women). Or you could try something more feminist-like, explaining you really don't like to be objectified by your appearance, or say having big breasts makes you feel like a bimbo, or something like that.
Or just say "Thanks, but I'd really rather not look like that." If they ask why, answer vaguely with "I just don't." Most people are polite enough to not pry further, and there are a lot of explanations for it they can imagine that do not involve gender.
