“Now, would you rather that a Googolplex people got dust specks in their eyes, or that one person was tortured for 50 years? I originally asked this question with a vastly larger number - an incomprehensible mathematical magnitude - but a googolplex works fine for this illustration.”
Yudkowsky thinks you should prefer the man be tortured for fifty years. Carlos disagrees. Essentially, Carlos thinks that there is a qualitative difference between pain/suffering and discomfort and pain seems to take something like lexical priority over discomfort- any amount of discomfort doesn’t outweigh pain.
Let me give his argument in abbreviated form in his own words:
“Yudkowksy argues that the math shows that, contrary to intuition, it is worse for a googolplex people to get dust specks in their eyes than one person to get tortured for 50 years. If there were such a thing as utility, this would be true, as the negative utility of getting a dust speck in your eye would have to be very tiny indeed to not balloon into a massive number when multiplied by a googolplex. But this is not actually the case.”
"…no amount of dust specks add up to suffering. In a twist of serendipity, while I was writing this essay, I actually did get a dust speck in my eye, which was very lucky as it had been a while since that happened and I couldn’t remember how it felt. It was mildly uncomfortable for about 5 seconds then it was gone. It was so mild, I don’t really remember it all that well, in spite of it happening a few days ago."
“Naively, it can seem like adding up enough discomfort nets you suffering, but when you think of actual examples of ordinary sensory inputs becoming pain through increased intensity, like say, overeating, listening to loud music, or looking at the sun, the pain starts because the body is actually being damaged (yes, you can rupture your stomach by overeating), not because the stimulus made it past some threshold and became pain. Pain is created through a different kind of input entirely, not through the addition of things that are not pain. It’s not like cranking the volume on a radio, it’s like tuning into a different channel, or more like getting the ordinary programming interrupted because of an emergency broadcast.”
There are really two key claims here
It’s not the case that pain is the same thing as discomfort, just at a higher magnitude.
Nor is there some third thing, utility, some common currency, through which pain can be traded for discomfort. Pain is worse, and that’s all we can say.
But I disagree. I’m dubious about 1, is it really plausible that for every single thing we call pain and every single thing we call discomfort, not a single one of them is capable of shading up from discomfort to being pain? There are, after all, many types of both pain and discomfort. I’ll grant 1 though for the sake of argument.
But I wholly reject 2.
We know that, at some level within an individual, discomfort is interchangeable with pain as far as people or the vast majority of people see it. To see this, consider the question, would you rather be tortured for a minute or keep getting dust specks in your eye just about continuously for fifty years? Assume these continuous dust specks won’t harm your eyes.
I can assure you, so long as the torture leaves no lasting mutilation, you should prefer the torture. I’ve been (consensually) waterboarded, which is milder than some forms of torture, but it is -you can trust me on this- definitely torture (Ron de Santis should be tried for crimes against humanity). Now let me be clear, I am in no way trying to make myself sound like a badass. I know torture under “naturalistic” conditions is vastly worse, but I feel I’ve had enough awful things happen to me (and you probably have to) that you can confidently make the tradeoff between a minute of torture and fifty years of continuous dust specks. I don’t really think this one is up for debate.
Thus even if we grant the premise that discomfort is qualitatively distinct from pain, it is still possible to rationally tradeoff pain against discomfort within an individual, so the author's premise seems wrong to me. As I’ll discuss later, perhaps you might think that for moral reasons we shouldn’t trade off the torture of a man for fifty years with a Googolplex of dust specks, but the claim there’s not even a shared currency here seems false.
Qualitative differences in function
Now I imagine Carlos might try to save their premise by saying that there's something qualitatively different about specks in the eye for fifty years because at that level it's going to interfere with the functioning of a person, whereas a once-off speck doesn't. Thus we can trade off fifty years of dust specks in eyes, against torture, but not a Googplex of singular dust specks against fifty years.
What this misses is that, over a Googleplex people, there will be significant functional differences made by the specks. Countless in that Googleplex will never have that crucial breakthrough or insight because that speck interfered with their train of thought, or will have a crucial day turn out to be rubbish because the speck was the straw that broke the camel's back and their capacity to cope. How many loves will go unkindled, how many stars uncrossed, how many pens dry of the ink of masterpieces because of those accursed specks?
There's a general important point here- even if higher order "overall" functioning is what matters, and not moments of discomfort or even pain- and I am somewhat sympathetic to that view- moments of discomfort still negatively affect overall functioning with some probability, and so can still be traded off with them.
A purely normative argument courtesy of Tim Williamson
Another way Carlos might go is to make the argument purely normatively. Regardless of whether or not a common currency of utility exists, we should refuse to tradeoff certain kinds of disutility for others, in particular, we should reject putting people in great pain in order to make a large number of people’s condition slightly better. I think it was the philosopher Tim Williamson who first tried an argument like this with the Soccer case. A football game is being broadcast over a country. If the broadcast stops, the game won’t stop, and it won’t be played again later. Millions of people are watching. A repairman goes up to the broadcast antenna to fix some minor issue, on the way up he slips and falls and breaks his leg very painfully. He’s in great pain but is at no risk of death or permanent injury. Getting him down will require stopping the broadcast. Should you stop the broadcast, or should you leave him to suffer 90 minutes or so of agony so a million people can watch their beloved game?
My initial reply to this argument would be to say, again, that what it misses are tipping points in human lives. Let’s say five million people are watching. If that soccer match is genuinely making people happy it’s not at all unlikely that for one of those five million people watching it, the rush of joy they get is the thing that stops them from suiciding in the next few weeks. About 1 in 10,000 people kill themselves each year, so in expectation about 500 viewers of the game will kill themselves within a year. If you’ve ever seen how much soccer fans seem to enjoy soccer it would not be at all surprising if for one of those 500 people, a delightful game of soccer was the thing that kept them going. How many families might be bought together? How many bouts of depression stomped before they can start? How many friendships kindled by that game of soccer?
Okay, but what if you said “Sure, things might be like that in real life, but for the sake of this thought experiment I want to specify that there will be no change in their overall functioning. We’re purely trading off the experience of a million people watching soccer against one guy writhing in agony for ninety minutes. Assume this has no greater impact on their lives than that.”
My first reply is that this specification is so weird and unnatural that it will distort our feelings in unforeseen ways.
My second reply is that this cuts both ways. If we have a full assurance that the repairman will suffer zero overall consequences to his life- just 90 minutes of agony and then it’s done- no further ill effects whatsoever, once that assurance is given in full, I think it becomes a lot more tempting not to disrupt the broadcast. Like most readers of this piece, I’ve been dreadfully sick and in a great deal of pain at some point in my life and so long as I had the assurance that this too would pass in 90 minutes then life will be back to normal I wouldn’t dream of making 5 million people much less happy in that moment to stop it.
Now to be clear I think, for a number of practical reasons, if this situation occurs, then TV broadcasters should be legally obliged to stop the broadcast. For the most part, I don’t want people making utility calculations in situations like this, I want nice simple rules like- “It’s illegal to leave people suffering in great pain when you’re their employer”.
Okay, but does utility actually exist?
We know that utility exists in sort of revealed, behavioral sense, people are capable of trading off chances of almost all sorts of goods and bads against each other, but does it actually, like, exist in the brain? Surely it’s not so simple.
Shockingly the answer is maybe.
The dopaminergic reward system seems to function very much like this on my understanding, with changes in Dopamine reflecting jumps or falls in our evaluation of how rewarding our situation is, as Caplin and Dean write:
“The neurotransmitter dopamine has been found to play a crucial role in choice, learning, and belief formation. The best developed current theory of dopaminergic function is the “reward prediction error” hypothesis—that dopamine encodes the difference between the experienced and predicted “reward” of an event.”
Here’s a bit from my thesis on the subject:
Stauffer, Armin and Schultz (2014) found that the dopaminergic system reflected the revealed cardinal utilities of monkeys making decisions under risk: “Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices”. Hybrid approaches that bridge psychology, biology and formal constructions on behaviour seem like a promising direction for future research- one that could solve, among other problems, the embarrassment of riches issue- although this is not the approach we are developing in this thesis. A Biorvix preprint by Matsumori et al. (2021) focusing on interpersonal comparision further develops a similar and exciting line of research, using a much larger sample of human subjects they deployed: “A method based on brain signals that correlated with changes in expected utility weighted by subjective probability. The signal was larger for participants with lower household incomes than for those with higher household incomes and their ratio coincided with the estimates by “impartial spectators””- In other words, they found that estimates by third parties of comparative utility changes between persons corresponded to the difference in magnitude of the activation of brain regions associated with reward.
So maybe your brain does have a common currency of “goodness” and “badness”, “reward” and “punishment” which it keeps track of in a very simple way- a very direct implementation of utility.
Expected value versus expected utility
I thought- even though it’s not really the topic of this piece- that I should say something briefly on Carlos’s critique of expected utility. Specifically, he argues It’s not risk-averse enough because it would counsel us to save 10 billion lives with 1% odds rather than 1 million lives with certainty.
But this isn’t necessarily true. The point of expected utility (or expected value) is to maximize expected utility, which is not necessarily the expected amount of some good, like lives. In this case, for example, because 10 billion deaths with no survivors might wipe everyone out if you think a lot of the value in present generations is in thinking there are future generations, you might think 1 million people surviving has, say, 50% of the value or utility of saving 10 billion, because both ensure future generations exist. The key question here is: is expected utility linear in the number of lives saved? The answer isn’t necessarily yes, especially for large numbers of lives.
Carlos notes that investors aren’t counseled to maximize expected value. The reason is a verbal difference. By expected value, investors typically mean money, however, utility is concave in money (and most goods), especially at large amounts of money, it looks like the one in the middle:
Addenda
Martin writes:
Okay, I decree that the dust specks have exactly zero knock-on effects; in each of the countless individual cases the total effect on utility at any timescale is the temporary discomfort alone. Assume a gazillion combo dust speck/amnesia events if you want. What makes Yudkowsky's argument interesting in the first place is that he (and presumably you) would still choose fifty years of torture at some number of these dust specks. You say that "there's a general important point here- even if higher order 'overall' functioning is what matters, and not moments of discomfort or even pain- and I am somewhat sympathetic to that view- moments of discomfort still negatively affect overall functioning with some probability, and so can still be traded off with them." But make that probability zero and nothing about your argument should change. There's still some number of dust specks that wins the utility comparison.
Later, you respond to a similar argument with "this cuts both ways. If we have a full assurance that the repairman will suffer zero overall consequences to his life- just 90 minutes of agony and then it’s done- no further ill effects whatsoever, once that assurance is given in full, I think it becomes a lot more tempting not to disrupt the broadcast." It doesn't have to cut both ways. Just construct the hypothetical like I did so that it only cuts one way, this doesn't change the utility math at all it just makes the necessary number a little bigger. By admitting that overall functioning plausibly matters and resorting to small probabilities of small impacts having large knock-on effects, you're undermining the original argument. If utility is a perfect common currency, truly meaningless tiny events with the measliest bit of negative utility and zero further negative consequences will eventually outweigh Hell itself if the numbers add up. And that's why this argument is fundamentally wrong. Yes, utility exists, but there must be some lexical priority consideration as well.
I reply:
Sure, if you go that way I just say you shouldn't maximize utility, you should maximize something more like "overall functioning" or eudaimonia, which is consistent with the kind of (approximate) utilitarian that I am, focused on something more like a eudaimonic or objective list theory of the good. I think I state above that I am sympathetic to such approaches to utilitarianism, and I previously covered my view here:
https://substack.com/inbox/post/128428596
The main purpose of this article is to argue utility does exist, but I agree, in the limit, there is an argument that utilitarianism should prefer something other than utility as its yardstick of wellbeing. However, as far as arguments that prove you should diverge from classic utilitarianism to something more like Eudaimonia utilitarianism, I consider this one not especially strong, since unlike in, say, the "tiling the universe" case, the intuition is far from overwhelming, and it never really applies in the real world, as there's always the possibility of making a functional difference with a small negative intervention here.
Okay, I decree that the dust specks have exactly zero knock-on effects; in each of the countless individual cases the total effect on utility at any timescale is the temporary discomfort alone. Assume a gazillion combo dust speck/amnesia events if you want. What makes Yudkowsky's argument interesting in the first place is that he (and presumably you) would still choose fifty years of torture at some number of these dust specks. You say that "there's a general important point here- even if higher order 'overall' functioning is what matters, and not moments of discomfort or even pain- and I am somewhat sympathetic to that view- moments of discomfort still negatively affect overall functioning with some probability, and so can still be traded off with them." But make that probability zero and nothing about your argument should change. There's still some number of dust specks that wins the utility comparison.
Later, you respond to a similar argument with "this cuts both ways. If we have a full assurance that the repairman will suffer zero overall consequences to his life- just 90 minutes of agony and then it’s done- no further ill effects whatsoever, once that assurance is given in full, I think it becomes a lot more tempting not to disrupt the broadcast." It doesn't have to cut both ways. Just construct the hypothetical like I did so that it only cuts one way, this doesn't change the utility math at all it just makes the necessary number a little bigger. By admitting that overall functioning plausibly matters and resorting to small probabilities of small impacts having large knock-on effects, you're undermining the original argument. If utility is a perfect common currency, truly meaningless tiny events with the measliest bit of negative utility and zero further negative consequences will eventually outweigh Hell itself if the numbers add up. And that's why this argument is fundamentally wrong. Yes, utility exists, but there must be some lexical priority consideration as well.
For me, I don't believe in utility for a few reasons, including:
(1) I don't think preferences are transitive (which is necessary for there to be a common currency)
It is easy to construct hypothetical and arguably realistic scenarios where people have logical rules in applying preferences that are not transitive, and I find it psychologically plausible that people could (unconsciously) apply these rules. Basically, all you need to do is add in more than one dimension/quality that is relevant to the person's preference, which sounds super plausible to me.
(2) I think how desirable a specific state of affairs is, is a relative judgement that depends on a number of factors including the identity of the evaluator, the time and context of the evaluation, the time and context of the state of affairs being rated, etc. So I don't think there is an eternal, single quantity to how desirable a state of affairs is. For an example, think of that famous story with the person breaking their leg and then not being drafted and then...