Allegheny Family Screening Tool
Predictive Risk Modeling in Child Welfare in Allegheny County
Since August 2016, the Allegheny County Department of Human Services (DHS) has used the Allegheny Family Screening Tool (AFST) to enhance our child welfare call screening decision making process with the singular goal of improving child safety.
The AFST is a predictive risk modeling tool that rapidly integrates and analyzes hundreds of data elements for each person involved in an allegation of child maltreatment. The tool can rapidly integrate and analyze these data, housed in the DHS Data Warehouse, and create a synthesized visualization of the information. The result is a ‘Family Screening Score’ that predicts the long-term likelihood of future involvement in child welfare. By combining the insight gained through the score with other traditionally gathered information, a better prediction can be made of the long-term likelihood that the child will need to be removed from the home in the future.
According to the algorithm, the higher the score, the greater the chance of future out-of-home placement. When the score is at the highest levels, meeting the threshold for ‘mandatory screen in,’ the allegations in a call must be investigated. In all other circumstances, the information summarized by the score does not replace clinical judgment but rather provides additional information to assist in the call screening decision making process. The Family Screening Score is not used to make investigative or other child welfare decisions and is not shared beyond call screening.
Background
Utilizing Integrated Data from the DHS Data Warehouse
Since 1998, DHS has used its DHS Data Warehouse to collect and store confidential data related to individuals who receive services through DHS.
In 2014, with support from the Richard King Mellon Foundation, DHS launched a highly competitive Request for Proposals (RFP) process to solicit proposals(PDF, 2MB) to explore how we could better use these data to improve decision making through predictive risk modeling.
The contract was awarded to a team from Auckland University of Technology (AUT) led by the co-director of the Centre for Social Data Analytics, Rhema Vaithianathan, and including Emily Putnam-Hornstein from the University of Southern California, Irene de Haan from the University of Auckland, Marianne Bitler from the University of California – Irvine, and Tim Maloney and Nan Jiang from AUT.
Over the next months, DHS and the AUT team explored various applications of predictive analytics and decided to focus initial efforts on child welfare and develop a tool that would assist in decision making when allegations of maltreatment are received.
Predictive Analytics and Child Welfare
The video below explains our partnership with Rhema Vaithianathan in the development and implementation of a predictive risk model for child welfare.
Prior to August 2016, any allegation of maltreatment (a referral) received at the child abuse call center required staff receiving the referrals – and their supervisors – to manually access a myriad of data and information to help decide whether or not to investigate the allegation (‘screen in’ and investigate or ‘screen out’ and offer relevant community resources).
In the absence of standardized protocols for using the data or for systematically weighting the information, an analysis found that 27% of highest risk cases were being screened out and 48% of the lowest risk cases were being screened in.
We reasoned that by using technology to gather and weigh all available pertinent information we could improve the basis for these critical decisions and reduce variability in staff decision-making.
The Allegheny Family Screening Tool
The design and implementation of the AFST was a multi-year process that included careful procurement, community meetings, a validation study, and independent and rigorous process and impact evaluations. In addition, the resultant model was subjected to an ethical review prior to implementation. Tim Dare, Professor of Philosophy at the University of Auckland and Eileen Gambrill, Professor of Child and Family Studies at the University of California-Berkeley conducted the independent ethical analysis. Their report, Ethical Analysis: Predictive Risk Models at Call Screening for Allegheny County, concluded the tool is ethically appropriate, particularly because its accuracy exceeded the alternatives at the time and there would be ethical issues in not using the most accurate measure. DHS issued Ethical Analysis: Predictive Risk Models at Call Screening for Allegheny County - DHS Response(PDF, 75KB) thereafter.
Details of this process and data related to the implementation of the screening tool are provided in the AUT report Developing Predictive Models to Support Child Maltreatment Hotline Screening Decisions - Allegheny County Methodology and Implementation, released in May 2019.
Evaluation
Formal evaluation of the Allegheny Family Screening Tool implementation process and impact began in 2015, following a DHS-issued RFP for assessments(PDF, 495KB) in these areas. Two separate contracts were awarded.
Hornby Zeller Associates, Inc., a public sector evaluation, research and consulting firm, was awarded the task of examining the process by which the tool was implemented. The firm analyzed how the tool has changed the experience of call screeners, the practice and policy implications of its use, and perceptions and reactions to the tool. Their report, Allegheny County Predictive Risk Modeling Tool Implementation: Process Evaluation,(PDF, 735KB) was published in January 2018.
Jeremy Goldhaber-Fiebert, Assistant Professor of Medicine at Stanford University, examined the impact of the tool on the accuracy of decisions, disparity rates, overall referral rates and workload and more. A summary of the impact evaluation(PDF, 190KB) and the full evaluation report(PDF, 2MB) were published in April 2019.
We continue to refine the model based upon these evaluation results, frequent analysis and ongoing feedback from call screening staff.
AFST Examined
DHS-generated Content
What happens when human workers oversee algorithmic tools?
May 24, 2022
CMU researchers respond to inaccuracies in an AP report about their work analyzing the AFST.
DHS response to the Associated Press article “An algorithm that screens for child neglect raises concerns”(PDF, 157KB)
May 9, 2022
As part of “Tracked,” a series of articles on the government’s use of data, reporters Garance Burke and Sally Ho examined how the AFST impacted racial disparity in screening child welfare calls. Their article contained several points that we feel were inaccurate, and we offer our clarifications here.
DHS Response to 'Automated Inequality' by Virginia Eubanks(PDF, 236KB)
January 31, 2018
Ms. Eubanks set out to examine the development and use of the Allegheny Family Screening Tool (AFST). Consistent with other decisions we have made to promote rigor, transparency and accountability in our implementation of this new tool, we invited the author into our agency. Unfortunately, her piece has numerous inaccuracies and several key points require correction.
Allegheny Screening Tool - Frequently Asked Questions and Answers(PDF, 774KB)
DHS Office of Data Analysis, Research and Evaluation, July 2017
Developing Predictive Risk Models to Support Child Maltreatment Hotline Screening Decisions
DHS Office of Data Analysis, Research and Evaluation, March 2017
Local Reaction
We will use all resources to keep children safe
Pittsburgh Post-Gazette, March 23, 2018
This letter to the editor responds to letter from child welfare advocate Richard Wexler.
An agency that works, helping kids and their families (op-ed)
Pittsburgh Post-Gazette, February 11, 2018
National Reaction
Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals - Evidence from the Allegheny Family Screening Tool
JAMA Pediatrics, August 2020
Want Less-Biased Decisions? Use Algorithms
Harvard Business Review, July 26, 2018
Can Big Data Help Save Abused Kids?
Reason.com, February 2018
This article discusses efforts to use predictive analytics in preventing child abuse.
Can an Algorithm Tell When Kids are in Danger?
The New York Times, January 2, 2018
This article takes an in-depth look at Allegheny County’s use of predictive analytics. This article also appeared in the Irish Examiner.
Actionable Intelligence
University of Pennsylvania’s Actionable Intelligence for Social Policy
This video highlights the Allegheny Family Screening Tool as one of the projects across the country that use integrated data systems (IDS) to improve social programs.
Predictive Analytics in Child Welfare
U.S. Department of Health and Human Services Office of the Assistance Secretary for Planning and Evaluation, November 2017
This report examines the potential benefits and pitfalls of predictive analytics and provides advice for progress.
How Machine Learning Can Improve Public Sector Services
The Regulatory Review, October 2017
This article discusses how Allegheny County government digitized its records and uses big data analysis to improve health and human services.
Predictive Analytics in the Child Welfare System, Starting with the Basics
Alliance for Racial Equity in Child Welfare and the Center for the Study of Social Policy
This webinar discuss the development and use of predictive analytics in child welfare.
Using Integrated Data Systems to Improve Case Management and Develop Predictive Modeling Tools(PDF, 156KB)
Annie E. Casey Foundation, 2017
Four reports explore the value of using Integrated Data Systems to improve outcomes for individuals and families.