Discussion about this post

User's avatar
Ned Oat's avatar

Okay, I decree that the dust specks have exactly zero knock-on effects; in each of the countless individual cases the total effect on utility at any timescale is the temporary discomfort alone. Assume a gazillion combo dust speck/amnesia events if you want. What makes Yudkowsky's argument interesting in the first place is that he (and presumably you) would still choose fifty years of torture at some number of these dust specks. You say that "there's a general important point here- even if higher order 'overall' functioning is what matters, and not moments of discomfort or even pain- and I am somewhat sympathetic to that view- moments of discomfort still negatively affect overall functioning with some probability, and so can still be traded off with them." But make that probability zero and nothing about your argument should change. There's still some number of dust specks that wins the utility comparison.

Later, you respond to a similar argument with "this cuts both ways. If we have a full assurance that the repairman will suffer zero overall consequences to his life- just 90 minutes of agony and then it’s done- no further ill effects whatsoever, once that assurance is given in full, I think it becomes a lot more tempting not to disrupt the broadcast." It doesn't have to cut both ways. Just construct the hypothetical like I did so that it only cuts one way, this doesn't change the utility math at all it just makes the necessary number a little bigger. By admitting that overall functioning plausibly matters and resorting to small probabilities of small impacts having large knock-on effects, you're undermining the original argument. If utility is a perfect common currency, truly meaningless tiny events with the measliest bit of negative utility and zero further negative consequences will eventually outweigh Hell itself if the numbers add up. And that's why this argument is fundamentally wrong. Yes, utility exists, but there must be some lexical priority consideration as well.

Expand full comment
N3's avatar

For me, I don't believe in utility for a few reasons, including:

(1) I don't think preferences are transitive (which is necessary for there to be a common currency)

It is easy to construct hypothetical and arguably realistic scenarios where people have logical rules in applying preferences that are not transitive, and I find it psychologically plausible that people could (unconsciously) apply these rules. Basically, all you need to do is add in more than one dimension/quality that is relevant to the person's preference, which sounds super plausible to me.

(2) I think how desirable a specific state of affairs is, is a relative judgement that depends on a number of factors including the identity of the evaluator, the time and context of the evaluation, the time and context of the state of affairs being rated, etc. So I don't think there is an eternal, single quantity to how desirable a state of affairs is. For an example, think of that famous story with the person breaking their leg and then not being drafted and then...

Expand full comment
3 more comments...

No posts