From Humans to AIs: The Unsettling Shift in AI Training and Its Consequences
This piece is important.
I’m a PhD student in natural language processing (NLP), the area behind AI models like ChatGPT and Siri.
I’m writing this article because I want to warn people now about a trend I’m seeing in NLP research where the ethics papers describing how bad this trend is will likely only come out months or years after this piece is released.
We’re replacing humans with AIs.
What do I mean by that?
Previously, if you wanted to use an AI model to make predictions, you would train that model on predictions made by humans, and teach the model to imitate those predictions. For example, ChatGPT was trained on the internet, which, for the most part, is made up of human text.
We’re beginning a new era where instead of training our AI models to imitate humans, we’re training them to imitate other AIs. So, if you’re building a text generation AI like ChatGPT, instead of training it on the internet, you might instead train it on ChatGPT’s outputs.
Why is this bad?
- AI systems are often biased and reflect the opinions of the majority, rather than reflecting the reality of diversity [Santurkar et al., Prabhakaran et al.]. So, by training future AI systems on existing AI systems’ outputs, we increase that bias and lack of diversity.
- Humans change over time. If we train a new AI on data produced by an old AI, the new AI will reflect the (likely majority) opinions of the people whose opinions the old AI was trained on.
My lab is currently conducting research to help understand how much generative AIs (like ChatGPT) tend to reduce diversity. However, in the meantime, while this research is being done, papers are coming out that show that for some tasks, AI models are able to replicate the majority opinions of humans very well, such that they could be used as a cheaper replacement for humans [Gilardi et al., He et al.]!
What disturbs me about these papers is
- The two studies cited literally ignore any non-majority opinions. The only thing they test is how good the models are at replicating the majority opinion among humans. They explicitly remove any data that does not conform to the majority opinion when measuring the accuracy of their models.
- The papers are extremely practical and implementable. One shows a technique for how to get AI models to better imitate majority opinions [He et al.], and the other one talks about pricing and how using an AI is cheaper [Gilardi et al.] than humans. These are papers that could already be implemented in existing websites or apps.
This is not good for humanity, and I’m worried.
What can we do about it?
If you’re going to train your AI model on outputs generated by other AIs, please only do this for datasets where humans have an extremely high agreement on the answer.
For example, if you’re training an AI to have “proper” grammar (according to some style guide), or to do math problems, or answer factual questions, then I think it’s generally fine to train on AI-generated data, because if humans have disagreeing opinions in those cases, usually someone is right and the others are wrong.
However, in cases like determining whether something is hate speech, or trying to figure out if a particular action is ethical, please do not train your AI to mimic another AI’s answers. Your AI will end up biased towards the majority opinion of whatever people the original AI trained on, rather than reflecting the diverse opinions of the people your AI will likely be interacting with.
I don’t think it’s good that Gilardi’s paper explicitly points out how often humans disagree on their tasks and that the AI doesn’t reflect that, and views that as a positive attribute of the AI [Gilardi et al.]. I think the authors’ assumption is that being able to just model the majority opinion is a step forward for AI. From a technical standpoint, this may be true. But from an ethical standpoint, I believe that ChatGPT’s lack of diversity of responses may be more of an indication of bias and a lack of perspective, rather than an indication of correctness or “accuracy”. (It may also be that the authors view their tasks as objective; in that case, I disagree with them.)
Summary
In the coming months and years, numerous AI systems will be trained on data created by other AIs, rather than human-generated data. It is crucial to address the potential biases and lack of understanding of humanity’s diversity and dynamics in these systems now, rather than relying solely on AI-generated data that may not accurately represent our diverse world.
As always, I love hearing different perspectives and advice, so if you disagree or have thoughts as to how this piece could be made better, please let me know in a comment!
(The title, preview subtitle, and summary section were generated with the aid of GPT-4.)
