Human vs. AI Decision-Making: Insights from Professor Kosuke Imai's Talk

Professor Imai introduced a new framework to evaluate a critical question: does AI help humans make better choices, or do human-alone or AI-alone systems perform better? Drawing on insights from a randomized controlled trial focused on pretrial risk assessment in the criminal justice system, he examined specific conditions under which AI recommendations enhance decision-making versus situations where they might prove counterproductive. The talk shed light on AI's role in high-stakes decisions, such as bail determinations, making it relevant to researchers, policymakers and anyone interested in the future of AI and society. His approach employs:

• Standard classification metrics based on baseline potential outcomes to measure decision-making accuracy
• A single-blinded treatment assignment, where AI-generated recommendations are randomized across cases
• A comparative analysis of three decision-making approaches: human-alone, human-with-AI, and AI-alone

The use of Artificial Intelligence, or more generally data-driven algorithms, has become common in today's society. Yet, in many cases, especially when stakes are high, humans still make the final decisions.

Importantly, the methodology developed by Professor Imai extends beyond the immediate study context. The research also identifies conditions under which AI recommendations should be provided to human decision-makers, and when such recommendations should be followed.

When applying this methodology to their randomized controlled trial evaluating a pretrial risk assessment instrument (PSA-DMF), Professor Imai and his team discovered that risk assessment recommendations did not improve the classification accuracy of a judge's decision to impose cash bail. Furthermore, they found that replacing a human judge with algorithms, specifically the risk assessment score and a large language model, led to worse classification performance overall.

Professor Imai and his team continue to explore extensions to their work, including multiple decision scenarios, joint potential outcomes, and dynamic decision processes. Their findings suggest that while AI tools have tremendous potential, they must be carefully implemented and evaluated, particularly in high-stakes decision environments. The team's open-source R package called "aihuman" provides researchers with tools to conduct similar analyses in other contexts.

Read more about the research behind the talk

Watch the talk here.

About Professor Kosuke Imai:

Kosuke Imai (pronounced Ksk) is a Professor in the Department of Government and the Department of Statistics at Harvard University. He is also an affiliate of the Institute for Quantitative Social Science. Before moving to Harvard in 2018, Imai taught at Princeton University for 15 years where he was the founding director of the Program in Statistics and Machine Learning. Imai specializes in the development of statistical methods and machine learning algorithms and their applications to social science research. His areas of expertise include causal inference, computational social science, and survey methodology. Imai leads the Algorithm-Assisted Redistricting Methodology Project (ALARM) and served as an expert witness for several high-profile legislative redistricting cases. In addition, he is the author of Quantitative Social Science: An Introduction (Princeton University Press, 2017). Imai served as the President of the Society for Political Methodology from 2017 to 2019.

His current research interests include: data-driven policy learning and evaluation, causal inference with high-dimensional and unstructured treatments (e.g., texts, images, videos, and maps), fairness and racial disparity analysis, algorithmic redistricting analysis, data fusion and record linkage, census and privacy.