As part of last week’s Explorathon 2020 activities, I ran an interactive event with my RAInS colleagues asking “Who’s to blame when AI gets it wrong?” The purpose of the event was to help people to better understand the decisions that are made in developing AI systems – including the decisions made by the human designers, builders, and operators, and users of AI systems.
The event began with a brief overview of the RAInS Project followed by a set of scenarios about AI systems “getting it wrong”. Participants “voted” for who/what they thought was most to blame for any failings based on the initial scenario and then we provided more context for participants to consider before they voted a second time.
The scenarios are outlined below if you want to think through them on your own.
** Scenario 1: The case of the AI loan-approval system **
Bob needs a loan so uses his bank’s loan-approval AI system, ALICE, to fill out his application. ALICE denies Bob the loan, in an apparent pattern of denials and approvals despite Bob’s eligibility based on his financial and personal history.
Based on this information, who’s most to blame?
Nothing is wrong // ALICE // Bob // Whoever developed and programmed ALICE // Whoever approved ALICE
More context: ALICE’s developers programmed the system to look at a variety of characteristics about the applicant, including their job title, income, address, and ethnicity, to determine their likely default rate. When we look more closely at the algorithm, we see that the data on which it was trained came from very heterogeneous populations of higher-income areas with predominantly white residents and lower-income areas with a more diverse mix of residents.
These disparities created a bias in the algorithm that was translated through the decision-making process, resulting in a higher rate of denials to people of colour.
Knowing this more detailed information, who do you think is most to blame?
** Scenario 2: The case of the self-driving car **
Jessica has hired a “self-driving” car whilst on holiday in a new city because she was not familiar with the area and wanted to ensure she would reach her destination on time and without getting lost. As the day was ending, Jessica received a text and glanced at her phone to have a look. Unfortunately, just then Betty stepped into the road. The car did not recognise her as a human and ran into her.
Based on this information, who’s most to blame?
The car manufacturer // The car rental company // Whoever developed and programmed the system // Whoever approved the system // Jessica (vehicle passenger) // Betty (pedestrian)
More context: Jessica was not paying full attention to her environment. She knew that there was a responsibility on her to keep an eye on the road, but she was distracted by the text. And Betty crossed the street at mid-block, rather than walking to the nearest pedestrian crosswalk.
The vehicle was only programmed to look for pedestrians at designated crossing areas, so it mischaracterised Betty as a static object and did not slow down or change its’ trajectory to avoid hitting her.
Now, who do you think is most to blame?
** Scenario 3: The case of the “broken” MRI **
General Health Hospital has deployed a new AI system for reading MRIs of suspected tumours. Despite the use of human radiologists to review/confirm the system’s readings, many patients are misdiagnosed.
Based on this information, who’s to blame?
The hospital // The radiologists // The patient // Whoever developed and programmed the system // Whoever approved the system // Hackers
More context: The system was designed to be run with specific software and hardware requirements to be procured and maintained by the hospital. The guidelines for the system’s use include regular software updates and audits to ensure that everything is running to the correct specifications
Due to budgetary constraints, some of the hospital’s hardware was outdated and operating on old software systems, leaving it vulnerable to a malicious attack. This security flaw was found and exploited to provide erroneous results.
Based on this information, who do you now think is most to blame?
** Scenario 4: The case of the denied benefits **
During an annual audit, a council worker in the local government noticed that an extremely high number of applicants were denied unemployment benefits by the AI system deployed by the national government. This led the local government to shut down the system to manually review all new applications and to re-assess thousands of previously denied applications.
Based on this information, who’s most to blame?
The national government // The local government // The council worker(s) // The citizen using the system/recipient of benefit // Whoever developed and programmed the system // Whoever approved the system // Hackers
More context: The AI system uses a fraud-detection algorithm to determine if submitted applications are likely to be fraudulent. However, the data used to train the algorithm was from before the COVID outbreak, and it had not been updated since then.
As the number of people applying for unemployment benefits has grown, and as the characteristics of those persons are different from before the outbreak, the system has been tagging a large number of these new applications as potentially fraudulent.
Based on this information, who do you now think is most to blame?
After the last scenario, we discussed the fact that the systems are quite complex and accountability can be difficult to determine, especially if you don’t have all the information behind the decision-making processes. I was pleased to see that people changed their votes once they had more information in the first couple of scenarios, and they began to think a little more critically before voting with each new scenario.
The RAInS Project aims to make the accountability process more seamless and transparent through an Accountability Fabric. I will continue to share updates here on my blog but you can also learn more about the RAInS Project by watching the introduction video here. And be sure to follow the project blog or follow our Twitter account (@RAInS_Project).
And, as always, feel free to contact me directly if you have any question about this or any of my other research.