Getty Images / WIRED
A flagship artificial intelligence system designed to predict gun and knife violence before it happens had serious flaws that made it unusable, police have admitted. The error led to large drops in accuracy and the system was ultimately rejected by all of the experts reviewing it for ethical problems.
The prediction system, known as Most Serious Violence (MSV), is part of the National Data Analytics Solution (NDAS) project. The Home Office has funded NDAS with at least £10 million during the last two years with the aim to create machine learning systems that can be used across England and Wales.
Advertisement
As a result of the failure of MSV, police have stopped developing the prediction system in its current form. It has never been used for policing operations and has failed to get to a stage where it could be used. However, questions have also been raised around the violence tool’s potential to be biased towards minority groups and whether it would ever be useful for policing.
The MSV tool was designed to predict whether people would commit their first violent offence with a gun or knife in the next two years. People who had already come into contact with the two police forces involved in developing the tool, West Midlands Police and West Yorkshire police, were given risk scores. The higher the score, the more likely they would be to commit one of the crimes.
Historic data about 2.4 million people from the West Midlands database and 1.1 million from West Yorkshire was used in the development of the system, with data being pulled from crime and custody records, intelligence reports and the Police National computer database.
But as NDAS was starting to “operationalise” the system earlier this year, problems struck. Documents published by the West Midlands’ Police Ethics Committee, which is responsible for scrutinising NDAS work as well as the force’s own technical developments, reveal the system contained a coding “flaw” that made it incapable of accurately predicting violence.
Advertisement
“A coding error was found in the definition of the training dataset which has rendered the current problem statement of MSV unviable,” a NDAS briefing published in March says. A spokesperson for NDAS says the error was a data ingestion problem that was discovered during the development process. No more specific information about the flaw has been disclosed. “It has proven unfeasible with data currently available, to identify a point of intervention before a person commits their first MSV offence with a gun or knife, with any degree of precision,” the NDAS briefing document states.
Before the error was found, NDAS claimed its system had accuracy, or precision levels, of up to 75 per cent. Out of 100 people believed to be at high risk of committing serious violence with a gun or knife in the West Midlands, 54 of these people were predicted to carry out one of these crimes. For West Yorkshire, 74 people from 100 were predicted to commit serious violence with a gun or knife. “We now know the actual level of precision is significantly lower,” NDAS said in July.
“Rare events are much harder to predict than common events,” says Melissa Hamilton, a reader in law and criminal justice at the University of Surrey, who is currently focussing on police use of risk prediction tools. Hamilton wasn’t surprised there were accuracy issues. “While we know that risk tools don’t perform the same in different jurisdictions, I’ve never seen that big of a margin of difference – particularly when you talk about the same country,” Hamilton says, adding the original estimations appeared to be too high based on other systems she had seen.
As a result of the flaw, NDAS reworked its violence prediction system and its results showed the significant accuracy drop. For serious violence with a gun or knife, the accuracy dropped to between 14 and 19 per cent for West Midlands Police and nine to 18 per cent for West Yorkshire. These rates were also similar whether the person had committed serious violence before or if was going to be their first time.
Advertisement
NDAS found its reworked system to be most accurate when all of the initial criteria it had originally defined for the system – first-time offence, weapon type and weapon use – were removed. In short, the original performance had been overstated. In the best case scenario the limited system could be accurate 25-38 per cent of the time for West Midlands Police and 36-51 per cent of the time for West Yorkshire Police.
The police’s proposal to take this system forward was unanimously refused. “There is insufficient information around how this model improves the current situation around decision making in preventing serious youth violence,” the ethics committee concluded in July as it rejected the proposal for the system to be further developed. The committee, which is a voluntary group consisting of experts from different fields, said it did not understand why the revised accuracy rates were sufficient and raised concerns about how the prediction system would be used.
“The Committee has expressed these concerns previously on more than one occasion without sufficient clarity being provided, and therefore as the project stands, it advises the project is discontinued,” the group said in its minutes. Committee members approached for this story said they were not authorised to speak on the record about the work.
Superintendent Nick Dale, the NDAS project lead, says those behind the project do “agree that the model cannot proceed in its current form” and points out that it has so far been experimental. “We cannot say, with certainty, what the final model will look like, if indeed, we are able to create a suitable model. All our work will be scrutinised by the ethics committee and their deliberations will be published.”
But multiple people who have reviewed the published NDAS briefings and scrutiny of the violence prediction system by the ethics committee say accuracy issues are only one area of concern. They say the types of data being used are likely to end up with predictions being biased, they have concerns with the normalisation of predictive policing technologies and cite a lack of evidence of the effectiveness of such tools. Many of these points are also reiterated in questions from the ethics committee to the NDAS staff working on the predictive systems.
“The core problem with the programme goes past any issues of accuracy,” says Nuno Guerreiro de Sousa, a technologist at Privacy International. “Basing our arguments on inaccuracy is problematic because the tech deficiencies are solvable through time. Even if the algorithm was set to be 100 per cent accurate, there would still be bias in this system.”
The violence prediction system identified “more than 20” indicators that were believed to be useful in identifying how risky a person’s future behaviour could be. These include age, days since their first crime, connections to other people in the data used, how severe these crimes were and the maximum number of mentions of “knife” in intelligence reports linked to them – location and ethnicity data were not included. Many of these factors, the presentation says, were weighted to give more prevalence to the newest data.
“There are a lot of categories, which have been proven in other areas of data analysis in the criminal justice system to lead to unequal outcomes,” says Rashida Richardson, a visiting scholar at Rutgers Law School who has studied data problems in predictive policing. “When you use age, that often skews most predictions or outcomes in a system where you’re more likely to include a cohort of people who are younger as a result of age just being one of the indicators used.” Hamilton agrees. She explains that criminal history factors are often biased themselves, meaning any algorithms that are trained upon them will contain the same issues if a human does not intervene in the development.
“We monitor bias and would not seek to deploy a model that contains bias,” says Dale, the NDAS project lead. “We are committed to ensuring interventions as a result of any model of this type are positive, aimed at reducing criminality and improving life chances, rather than coercive or criminal justice outcomes.”
“The main value in MSV is in testing the art of what is possible in the development of these techniques for policing,” Dale adds. “In doing so, it is inevitable that we will try things for whatever reason, but we are confident that as we progress, we are developing data science techniques that will lead to more efficient and effective policing and better outcomes for all of our communities.”
The current thinking of NDAS is that the predictive violence tool could be used to “augment” existing decision-making processes used by police officers when investigating people who are likely to commit serious violence. The violence prediction tool is just one that is being worked on by NDAS: it is also using machine learning to detect modern slavery, the movement of firearms, and types of organised crime. Cressida Dick, the head of London’s Metropolitan Police, has previously said police should look to use “augmented intelligence”, rather than relying on AI systems entirely.
However, issues of bias and potential racism within AI systems used for decision-making is not new. Just this week the Home Office suspended its visa application decision-making system, which used a person’s nationality as one piece of information that determined people’s immigration status, after allegations that it contained “entrenched racism”.
Last month, in the wake of the global Black Lives Matter protests, more than 1,400 mathematicians signed an open letter saying the field should stop working on the development of predictive policing algorithms. “If you look at most jurisdictions where there is some use of predictive analytics in the criminal justice sector, we don’t have evidence that any of these types of systems work, yet they are proliferating in use,” Richardson says.
Theses concerns are highlighted in the development of the violence prediction tool. Documents from the ethics committee show one unnamed member of the group saying the coding failure was a “stark reminder” about the risk of AI and tech within policing.
“In the worst case scenario, inaccurate models could result in coercive or other sanctions against people for which there was no reasonable basis to have predicted their criminality – this risked harming young people’s/anyone’s lives despite the clear warnings – however, it is good to see the team having evaluated its own work and identifying flaws from which to start again,” they said in March.
Despite the flaw in the violence predicting system, those who have reviewed it say the setup is more transparent than other predictive policing developments. “The committee’s advice is transparent, robust and has teeth,” says Tom McNeil, a strategic adviser to the West Midlands Police and Crime Commissioner. The fact that the ethics committee is asking pressing questions and getting answers is largely unheard of in the development of AI systems within policing – much of the development is usually done completely in secret with problems only emerging once they impact people in the real-world.“Just because something can be done computationally doesn’t necessarily mean that it’s always the best way to do it or that it should be done that way,” says Christine Rinik, the co-director of the Centre for Information Rights at the University of Winchester. “That’s why I think it’s so useful to have a process where these steps are questioned.”
Matt Burgess is WIRED’s deputy digital editor. He tweets from @mattburgess1
More great stories from WIRED
🚚 The French town that created its own Amazon
🦆 Google got rich from your data. DuckDuckGo is fighting back
😷 Which face mask should you buy? The WIRED guide
Advertisement
🔊 Listen to The WIRED Podcast, the week in science, technology and culture, delivered every Friday
👉 Follow WIRED on Twitter, Instagram, Facebook and LinkedIn
Get The Email from WIRED, your no-nonsense briefing on all the biggest stories in technology, business and science. In your inbox every weekday at 12pm sharp.
by entering your email address, you agree to our privacy policy
Thank You. You have successfully subscribed to our newsletter. You will hear from us shortly.
Sorry, you have entered an invalid email. Please refresh and try again.
