If you're really depressed, you have trouble doing anything, much less something that makes you happy.

I'd make a sarcastic, self-parodical "offended" comment along the lines of "stop making my condition a joke," but I think too many folks would take said comment seriously.