Chatbotocyplase 2, The botten-ing: A response to Scott Alexander on Chatbots
I’ve written this one pretty quickly because I’m busy this week, but I wanted to respond to Scott’s article because I find it interesting and because it seems like a good strategy for getting the word out about the blog. I apologize for errors of grammar etc.
Scott has an article “Mostly Skeptical Thoughts On The Chatbot Propaganda Apocalypse” replying to my article talking about the possibility of a social crisis caused by chatbots.
In my initial article I argued that it’s likely the internet will be flooded with chatbots, with written communication skills above those of the average person, serving nefarious political agendas of various sorts. Further, I argued has the potential to interact with AI caused job loss to trigger a crisis. I also tried to map out a leftwing political program intended to respond to this.
Summary of what I think of Scott’s take: I think Scott raises good points, and that my initial response might have reflected panic caused by the fact that, when I wrote the article, it seemed like I was the only person I knew IRL who was talking about machine learning. However, despite Scott’s arguments, I think there are genuine worries here, and a chatbotocyalpse is a live possibility.
I’ll focus on the first four points Scott makes, because I’m not quite sure what to make of point 5 (I’m unsure why we should think that armies of bitcoin scammer bots make us safer with respect to other bots?)
1. Celebrities as a model?
Scott writes:
In a sense, I’m living in the chatbot apocalypse already.
As a famous blogger, I live in a world where hordes of people with mediocre arguing skills try to fake shallow friendships with me to convince me to support things. But I can almost completely block it out. I do sometimes engage, but with people I know personally, or through some kind of vague community social proof, or because I’ve appreciated other original and interesting things someone has said. Other famous people have set their social media to only allow replies from people they follow, or from other bluechecks, or they use some other screening method that mostly works for them. I don’t think famous people get convinced of weird stuff more often than the rest of us, even though they labor under this disadvantage.
I’m not persuaded. The problems I see here are:
These bots are unlikely to be mediocre- they may start out with mediocre arguing skills, but there’s little reason to think they’ll stay that way. If the people who approached celebrities were charming and superhumanly patient they might have better luck.
If someone had succeeded in getting Scott to support something through social interaction in a tactful way, almost by definition Scott wouldn’t know it. It’s like those people who say “all the trans people I clock can’t pass, therefore no trans people pass”.
Famous people are generally quite sophisticated relative to the general population especially when it comes to playing social games and applying (and resisting) influence. The resilience of famous people to influence attempts may not apply to the general community.
Actually, it seems to me that famous people are unusually likely to become advocates for oddball causes, and plausibly this is at least in part because people proselytize at them (I think someone in the comments makes the same point).
Scott emphasizes arguing here. While doubtless some chatbots will argue, I suspect this will only be the tip of the iceberg of their persuasive tactics, which can be inhumanly patient and subtle compared with a singular human’s attempt to influence someone online.
2. Identity verification: our best chance
Then Scott goes on to argue that identity verification will be the silver bullet to stop these bots. I’m much more sympathetic to this argument:
Sites could ask for proof of humanity.
Yet I’m not entirely convinced this walled garden doesn’t have holes:
Naïve people will bring in bot written content from outside the walled gardens.
If the bots are good enough, even thoughtful people might read them in preference to humans because they’re more persuasive and informative, they will leave the walled gardens to seek them out.
Or they might seek them out because they’re more personable, erotic, entertaining or whatever. This ‘quality route’ could give a hook for uptake by sophisticated models controlled by wealthy actors.
Scammers and propogandists will find or pay a way in- such as paying for access to people’s accounts. Even if this requires the models to slow down their activity, it still gives them a route through.
People will commission content from chatbots to make themselves look smart and post it in these walled gardens. Assuming the best models are made by people who aren’t politically neutral, they could subtly bias the content they generate.
Really big players like the CIA might be able to subvert these companies to let their bots in. They might even be let in under the banner of “combatting disinformation” (you understand we don’t want to resort to these beastly methods, but our enemies… they are not so honorable).
People will seek out bots to help aggregate news and content into summaries and analysis, these summaries and analysis could be deliberately but subtly biased.
But I don’t find my counterarguments altogether persuasive (you can probably tell by the fact I’ve listed so many of them). The more I think about it, the more I start to think that my initial concerns that the bots will find a way in were overblown. This, more than anything else, makes me think Scott might be right. We shall see.
3. Reputational risk?
Scott argues that companies, individuals and governments will avoid using deceptive chatbots that try to win friendships because:
I learned that my Internet friend who I’d talked to every day for a year was actually a chatbot run by Pepsi trying to make me buy more Pepsi products, I would never buy another can of Pepsi in my life.
I think this may limit damage, but not by very much, because:
There are plenty of bad actors that don’t care much about their own reputation, they just want to damage the reputation of others, or otherwise muddy the waters. I think, for example, that certain states would be happy to turn the citizens of rival states against their government and sow cynicism and conflict- even if it risk a huge reputational backlash.
Even if it’s irrational to advocate for an ideology this way, it doesn’t take many people wrongly deciding it is rational for it to happen anyway. You only need a few people with resources to decide to go ahead with it anyway, whereas to prevent it you need a consensus among those with the resources to pull it off.
People will find subtle ways to push their agenda while with armies of electronic advocates while “playing within the rules” and not disguising themselves.
People will push it off onto deniable assets. “Oh, this isn’t CIA propaganda, it’s from the institute for freedom in Latin America. While their hearts are in the right place we simply abhor their tactics, you understand. Besides, the other side have plenty of scumbags who do this, you can’t hold it against us what a few extremists have done, that would be unfair.”
4. Anti-establishment propaganda or establishment propaganda?
Scott argues that we should be more worried about big government players and other members of the establishment using chatbots to spread the “official” line rather than disinformation:
Here’s a crazy thought: maybe the side that has 10x more people on it and 100x more support from the relevant actors is the one that will win the race to use this technology. You can already get an off-the-shelf chatbot that mouths liberal pieties; I’m not sure there are any that are much good at disinformation production. The liberal-piety-bots will be on version 10.0 before the first jailbroken disinformation-bot comes out.
So the establishment has a big propagandabot advantage even before the social media censors ban disinfo-bots but come up with some peaceful-coexistence-solution for establishment-bots. So there’s a strong argument (which I’ve never seen anyone make) that the biggest threat from propaganda bots isn’t the spread of disinformation, but a force multiplier for locking in establishment narratives and drowning out dissent. If this doesn’t worry you in the US, at least let it worry you in China, whose government already hires people to parrot propaganda on social media.
I agree completely. I 100% think that there will be numerous establishment-centrist-liberal bots. Given what I think of establishment NATO liberalism though, this isn’t much comfort.
My thoughts on Scott’s predictions:
1. If I ask ACXers in 2030 whether any of them have had a good friend for more than a month who turned out to (unknown to them) be a chatbot, or who they strongly suspect may have been a chatbot, fewer than 10% will say yes. I may resolve this by common sense if it’s obvious, or by ACX survey if it’s not: 95%
I certainly wouldn’t put this at 95%- perhaps 75% Still, I agree with the basic direction of the prediction.
However, I’m not sure this is a good indicator because: A) The SSC audience is unusually sophisticated and B) I think a bigger threat than outright ‘good friends’ who turn out to be chatbot is esteemed acquaintances. Think cool and trendy accounts which you interact with a bit, that seem to have a lot of followers, always have that snappy comebacks etc.
2. If I ask ACXers in 2030 to estimate what percent of people they follow on Twitter are secretly chatbots, the median answer will be 5% or less: 85%
I’m 50/50, the primary consideration being whether Twitter sets up a walled garden and how effective it is.
3. …1% or less: 60%
As above.
4. In 2030, an AI won’t be able to write blog posts as good as a 75th percentile ACX post, by my judgment. The AI will fail this task if there’s any kind of post I write that it can’t imitate - for example analyzing scientific data, or writing fiction, or reviewing books. It will fail this task it it writes fluently but says false things (at a rate higher than I do), eg if it makes up references. It doesn’t have to be able to coordinate complex multistep projects like the Book Review Contest: 45%
I’d say 25%. I think we’re really on the fast train now when it comes to language models.
5. In 2030, the majority of the top 10 blogs in Substack’s Politics category will be written by humans. Blogs where we’re not sure don’t count either way: 90%
Here I want to dissect the question:
A) I think there’s a very good chance that chatbots will become influential in other places on the internet, but Substack, for reputational reasons, won’t let them on.
B) A lot of the danger is in stuff that claims to be human but isn’t really, so how can we assess this?
C) Also the danger lies in blogs that claim to have a human author, and maybe once did, but in reality the human now merely curates AI content in their own style.
My final word on the topic of the Chatbotocalypse:
It’s almost certainly going to come down to some random factors we haven’t even dreamed up yet.
Two things to keep an eye on
On the topic of predicting the future, there’s two things I wanted to mention to readers. I don’t have enough content on these to write a
Are people sleeping on what's really amazing about "Multimodal Chain-of-Thought Reasoning in Language Models"?
A lot of people are very excited about this paper because it uses a cool method- reasoning, in words, via chain of thought, about stimuli that include both images and text to a conclusion.
But I haven't seen anyone yet draw attention (at least not very explicitly) to what may be its coolest feature- viz, even when images aren't involved, it far exceeds the performance of GPT-3.5 on the text problems, despite having about 1/250th the parameters. ( 95.26 v 74.68 when GPT uses CoT on text only problems- problems that theoretically it should have no inherent advantage over GPT in).
Comparing it to the same sized UnifiedQABase w/ CoT on the text questions we get a bounce of 66 versus 95% on the text problems.
The simplest theoretical interpretation would be to read this paper as suggesting that learning about language in a way that integrates images leads to deeper understanding, even when images aren't present at the inference stage? But honestly I’m out of my depth here. In the past I’ve argued that one can understand the world solely through text- but it makes a lot of sense that you can do so more efficiently if you integrate images.
I flagged this in the MLScaling subreddit and on a skeptical note: u/895158 commented: “Note that the authors themselves are not excited by this and barely mention it. I think it's because it's not a fair comparison: their model is fine-tuned on the training part of the dataset while the GPT3.5 baseline is not fine-tuned and uses 2-shot prompting. I'm happy to be corrected if I misunderstood”
Finetuned versus not finetuned has to be a big part of the explanation. However, it is not uncommon to finetune on datasets before attempting, and even in light of that I can’t help but think that beating a model 250X your size with 1/5th as many errors is very impressive.
It’s wholly possible I’ve simply misunderstood something, but it seems worth a closer look.
Panic is pointless, but we really need to be taking H5N1 more seriously
https://www.nytimes.com/2023/02/03/opinion/bird-flu-h5n1-pandemic.html
Re: smaller models in the CoT paper: You can think of large language models as generally coming in two varieties. The ones you read headlines about - GPT and friends - tend to be decoder-only architectures that are ridiculously large (often 100B or more parameters.) An organization needs to be either large, rich, or highly specialized to fine-tune a model like this; the point is that they're supposed to be extremely flexible and expressive even without fine-tuning, useful for zero-shot and few-shot learning. You won't see "fine-tuned GPT-3" in a chart like this because very few people have the resources to fine-tune it, even if OpenAI gives them permission to.
The non-GPT models in the chart, including the one you're looking at - Multimodal-CoT - are encoder-only or encoder-decoder architectures, much smaller - usually less than 1B parameters - such that it's relatively feasible for an independent researcher or small organization to fine-tune one. They are the unglamorous-but-still-kind-of-cutting-edge workhorses of commercial AI. They're nowhere near as flexible or expressive as GPT and kin, but they can be as good or better at a single, well-defined task; e.g. classifying whether tweets about a company's product express positive or negative sentiment, or selecting the sentence on a web page most likely to answer a user's query.
So, fine-tuned versus not-fine-tuned is the lion's share of the explanation here, and why it's more-or-less expected to see much smaller models outperform GPT-3 on certain kinds of tasks. Vanilla GPT-3 has the incidental ability to complete a wide variety of tasks, as a side effect of its pre-training on next-word-prediction, whereas Multimodal-CoT is lightweight, great at doing one thing, and useless for anything else.
Did you use ChatGPT to write your response? I'm guessing not. That seems as strong an argument against your thoughts as any.