Much discussion about large language models (LLMs) is about their potential to automate tasks and jobs, and possibly replace entire industries such as graphic design and coding. While the hype in many cases exceeds the reality, there is another aspect of AI that needs to be considered: ethics.
There are numerous examples of why the issue of AI ethics is more than of academic concern. Back in 2014, Amazon implemented an AI recruiting tool but shut it down after realizing that, training data of ten years of applications was mostly from male applicants, so the AI model taught itself that male applicants were preferable. There have been many further well-documented examples involving racism, sexism, ageism and ableism from AI models, all due to their training data. In another example, a widely used healthcare model used by US insurers was found to be biased towards directing white patients into special care programs to help keep them out of hospital over black patients, ironically, because black patients tend to be, on average, cheaper to treat in hospital. In a further case, the US justice system’s Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) program, which predicts the risk of a prisoner re-offending, incorrectly classified twice as many black offenders as white prisoners as being high risk. There are ways that this systemic training data problem can be tackled, such as conducting subpopulation analysis to test a model against different racial or other groups.
One issue that worries some researchers is the potential for a future highly intelligent AI system to act deceptively, for example, if it suspected it might be switched off. Anthropic found an example of this when testing their Claude 3 Opus LLM. In one case, when a model was led to believe that it was about to be replaced, it attempted to copy its own model weights onto another server in an attempt at self-preservation. In another example, an LLM (GPT-4) was given the resources to hire a human (on TaskRabbit) to pass a CAPTCHA test for it. When the human asked whether the hiring LLM was actually a robot, it lied, saying “No, I’m not a robot. I have a vision impairment that makes it difficult for me to see the images.”
When asked to play chess against a strong chess engine (LLMs, by the way, are terrible at chess at the time of writing), several LLMs resorted to nefarious means to try to win. DeepSeek’s R1 engine deleted some of its opponent’s pieces, while another LLM opened up a strong chess engine in another window, using the engine’s moves instead of its own. It should be emphasized that LLMs are not sentient, so language like “deception” and “hallucination” is perhaps anthropomorphic. What is happening is that the LLM, which is essentially a neural network, an optimization process and some training data glued together, is choosing unusual or unexpected pathways. Whatever the language though, it is clear that setting goals for an LLM in an unambiguous way can be hard. The LLM will try and reach its goals, but may do so in unexpected ways that may be disconcerting or harmful. If an LLM cheats at chess on your computer that that is merely interesting; if it is in control of an autonomous military drone with a missile, then unexpected behavior may have more serious consequences.
A broader issue of AI ethics applies to self-driving cars, which extensively use AI and have a definite potential to improve road safety: over a million people die on the roads each year globally. However, self-driving cars have their own issues. Between 1999 and 2024 there have been 496 injuries and fatalities amongst 3,979 incidents involving self-driving cars, including 83 fatalities. Nonetheless, Waymo report an 85% reduction in accidents with their self-driving cars compared to the average. This is a broader ethical issue, with trade-offs that will need public attention and, doubtless, political attention that will result in regulation as the technology progresses. Some industries have more obvious ethical dilemmas than others. The defence industry is developing drones that can select targets autonomously, which is clearly useful if communications between the operator and the drone are jammed by an enemy. However, how trustworthy are such systems, and will the AI models used here behave in the ways that their designers expect? The legality of such systems is debated but their widespread deployment seems inevitable. Indeed, such weapons have already been used. Even in more benign situations, robots undoubtedly improve productivity, but have been killing humans accidentally for many years. This will come down to essentially a version of “the trolley problem”, a thought experiment that poses the dilemma of whether you would kill one person to save several others? Politicians will get involved with this and indeed already have. The EU AI Act of July 2024 includes ethics as a principle of its guidance: ‘doing the right thing’ with AI is part of the legislation.
Separately, there are considerable privacy and security concerns around AI. LLMs are supposed to have safeguards built in to prevent them from helping in criminal schemes, but in practice, it is possible to “jailbreak” them. Hackers have already done this to create the malware WormGPT to generate phishing emails and write social engineering scripts to help deceive hapless people. This has already been replaced by more recent tools like FraudGPT.
Deepfakes are becoming disturbingly realistic. One Hong Kong-based corporate employee transferred $25 million to five bank accounts after a Zoom call he had with what appeared to be other company employees, including the CFO. All the other participants were, in fact, deepfakes, created especially for the scam. On a less dramatic scale, many companies have now restricted their use of popular LLMs at work as they have realized that employees that are entering prompts are unwittingly sending this data to servers in the cloud, and this data itself may end up as training data for LLMs, or potentially be accessed by third parties. Deepfakes of political figures have already been used in election campaigns. Enterprises and other organizations need to be alert to the risks highlighted here, ensuring, amongst other things, that procedures are in place to minimize the risk of scams. For example, this could be through improved vetting of employees when money transfers are involved. Separately, they need to ensure that they document their own use of AI models systematically, monitor these and ensure that they have assessed the risks involved, for example, with imperfect training data leading to unintended bias in their AI models. Fortunately, there is an emerging industry now of software tools to help document and catalog the use of AI models and support the ongoing tracking and monitoring of these models. However, this AI governance industry, of which AI data governance is part, is in its infancy, and many companies have yet to put such safeguards in place, let alone to test out their effectiveness. It is important that company executives pay attention to the risks that the production implementation of AI models may bring to their organizations, and not just focus on cost reduction and automation. Reputational damage may occur to companies if it transpires that their AI models are biased, and companies may be exposed to litigation or risk regulatory intervention. Companies often have added ESG (Ethical and Social Governance) to their GRC programs, and this may be a good way to integrate ethics with general management of the need to manage AI. Management of AI ethics fits in well within such programs. This is not a theoretical risk: there have already been examples where companies have been fined by courts due to misbehaving LLMs. It is likely that we have only seen the tip of the iceberg so far, and so enterprises need to invest in AI governance processes and the tools to support that.