Badly behaved artificial intelligence (AI) systems have a long history in science fiction. Way back in 1961, in the famous Astro Boy comics by Osamu Tezuka, a clone of a popular robot magician was reprogrammed into a super-powered thief. In the 1968 film 2001: A Space Odyssey, the shipboard computer HAL 9000 turns out to be more sinister than the astronauts on board think.
More recently, real-world chatbots such as Microsoft’s Tay have shown that AI models “going bad” isn’t sci-fi any longer. Tay started spewing racist and sexually explicit texts within hours of its public release in 2016.
The generative AI models we’ve been using since ChatGPT launched in November 2022 are generally well behaved. There are signs this may be about to change.
On February 20, the US Federal Trade Commission announced an inquiry to understand “how consumers have been harmed […] by technology platforms that limit users’ ability to share their ideas or affiliations freely and openly”. Introducing the inquiry, the commission said platforms with internal processes to suppress unsafe content “may have violated the law”.
The latest version of the Elon Musk–owned Grok model already serves up “based” opinions, and features an “unhinged mode” that is “intended to be objectionable, inappropriate, and offensive”. Recent ChatGPT updates allow the bot to produce “erotica and gore”.
These developments come after moves by US President Donald Trump to deregulate AI systems. Trump’s attempt to remove “ideological bias” from AI may see the return of rogue behaviour that AI developers have been working hard to suppress.
Executive orders
In January, Trump issued a sweeping executive order against “illegal and immoral discrimination programs, going by the name ‘diversity, equity, and inclusion’ (DEI)”, and another on “removing barriers to AI innovation” (which includes “engineered social agendas”).
In February, the US refused to join 62 other nations in signing a “Statement on Inclusive and Sustainable AI” at the Paris AI Action Summit.
What will this mean for the AI products we see around us? Some generative AI companies, including Microsoft and Google, are US federal government suppliers. These companies could come under significant direct pressure to eliminate measures to make AI systems safe, if the measures are perceived as supporting DEI or slowing innovation.
AI developers’ interpretation of the executive orders could result in AI safety teams being reduced in size or scope, or replaced by teams whose social agenda better aligns with Trump’s.
Why would that matter? Before generative AI algorithms are trained, they are neither helpful nor harmful. However, when they are fed a diet of human expression scraped from across the internet, their propensity to reflect biases and behaviours such as racism, sexism, ableism and abusive language becomes clear.
AI risks and how they’re managed
Major AI developers spend a lot of effort on suppressing biased outputs and unwanted model behaviours and rewarding more ethically neutral and balanced responses.
Some of these measures could be seen as implementing DEI principles, even as they help to avoid incidents like the one involving Tay. They include the use of human feedback to tune model outputs, as well as monitoring and measuring bias towards specific populations.
Another approach, developed by Anthropic for its Claude model, uses a policy document called a “constitution” to explicitly direct the model to respect principles of harmless and respectful behaviour.
Model outputs are often tested via “red teaming”. In this process, prompt engineers and internal AI safety experts do their best to provoke unsafe and offensive responses from generative AI models.
A Microsoft blog post from January described red teaming as “the first step in identifying potential harms […] to measure, manage, and govern AI risks for our customers”.
The risks span a “wide range of vulnerabilities”, “including traditional security, responsible AI, and psychosocial harms”.
The blog also notes “it is crucial to design red teaming probes that not only account for linguistic differences but also redefine harms in different political and cultural contexts”. Many generative AI products have a global user base. So this sort of effort is important for making the products safe for consumers and businesses well beyond US borders.
We may be about to relearn some lessons
Unfortunately, none of these efforts to make generative AI models safe is a one-shot process. Once generative AI models are installed in chatbots or other apps, they continually digest information from the human world through prompts and other inputs.
This diet can shift their behaviour for the worse over time. Malicious attacks, such as user prompt injection and data poisoning, can produce more dramatic changes.
Tech journalist Kevin Roose used prompt injection to make Microsoft Bing’s AI chatbot reveal its “shadow self”. The upshot? It encouraged him to leave his wife. Research published last month showed that a mere drop of poisoned data could make medical advice models generate misinformation.
Constant monitoring and correction of AI outputs are essential. There is no other way to avoid offensive, discriminatory or unsafe behaviours cropping up without warning in generated responses.
Yet all signs suggest the Trump administration favours a reduction in the ethical regulation of AI. The executive orders may be interpreted as allowing or encouraging the free expression and generation of even discriminatory and harmful views on subjects such as women, race, LGBTQIA+ individuals and immigrants.
Generative AI moderation efforts may go the way of Meta’s fact-checking and expert content moderation programs. This could have an impact on global users of US-made AI products such as OpenAI ChatGPT, Microsoft Co-Pilot and Google Gemini.
We might be about to rediscover how essential these efforts have been to keep AI models in check.
Judith Bishop has received funding from Creative Australia for a book on AI and human data. Until 2022 she led teams producing training data for global AI companies and US government research agencies.
