Current and former law students of the University of Cambridge organised a week-long competition to directly pit lawyers against artificial intelligence.
CaseCrunch, an AI startup founded by Rebecca Agliolo, Ludwig Bull and Jozef Maruscak, won the competition. With an accuracy of 86.6%, compared to the lawyers’ accuracy of 62.3%, CaseCrunch emerged victorious.
112 lawyers pre-registered to participate in the Lawyer Challenge, which ran from 20-27 October. They were presented with factual scenarios of payment protection insurance (PPI) mis-selling claims, and asked to predict "yes or no" as to whether the Financial Ombudsman would uphold the claim. The same factual scenarios were given to CaseCrunch: whoever had the highest accuracy, won. 775 predictions were submitted by the participants.
A Technology Judge and a Legal Judge independently verified the fairness of the competition. The Legal Judge was Dr Felix Steffek, University of Cambridge Lecturer in Law, and Co-Director of the Centre for Corporate and Commercial Law. The Technical Judge was Ian Dodd, UK Director of Premonition.
The knowledge jobs will go, the wisdom jobs will stay Ian Dodd, UK Director of Premonition
The factual scenarios were real decided cases from the Financial Ombudsman Service, published under the Freedom of Information Act. All identifying details - such as the name of the parties, case names and dates - were removed, leaving only the facts. Lawyers completed their predictions in an unsupervised environment, and were permitted to use all available resources. Participants were given links to the Financial Conduct Authority’s rules detailing the basis of an Ombudsman’s decision.
The competition was featured on the BBC website.