I find myself strategically using the "too busy" response out of politeness. Cheap zombie client returning from the past? Cropping and resizing 20,000 JPEGs? Sorry -- "I'm booked with work just now." It's a lot easier (and it feels nicer) than saying "pff, I don't want to do that."
This is why having a good network of people you can refer work to is important. Refer the good clients (in cases where you are, maybe, genuinely, too busy) to your talented friends. Refer the terrible clients to colleagues you don't like. ;)