Pretrial Risk Assessments


Read the Statement of Concerns ›

Read the Full Report ›

Six Things to Know about Algorithm-Based Decisionmaking Tools ›

Algorithm-Based Decisionmaking Tools: Frequently Asked Questions ›

We believe that jurisdictions should work to end secured money bail and decarcerate most accused people pretrial, without the use of “risk assessment” instruments.

The extraordinary measure of pretrial detention should be treated as a last resort and should only be imposed upon an accused person after they’ve received a thorough, adversarial hearing that observes rigorous procedural safeguards respecting individual rights, liberties, and the presumption of innocence.

In light of the concerns raised in this document, we urge jurisdictions to reconsider their use of risk assessment tools. Pretrial “risk assessment” instruments – although they may seem objective or neutral – threaten to further intensify unwarranted discrepancies in the justice system and to provide a misleading and undeserved imprimatur of impartiality for an institution that desperately needs fundamental change.

Where these tools are used, in order to reduce the harm they can cause we urge the following:

 

Principle 1


Pretrial risk assessment instruments must be designed and implemented in ways that reduce and ultimately eliminate unwarranted racial disparities across the criminal justice system. Those engaged in the design, implementation, or use of risk assessment instruments should also test ways to reduce the racial disparities that result from using historical criminal justice data, which may reflect a pattern of bias or unfairness.

 

Principle 2


Pretrial risk assessment instruments must be developed with community input, revalidated regularly by independent data scientists with that input in mind, and subjected to regular, meaningful oversight by the community. The particular pretrial risk assessment instrument chosen should be trained by, or at least cross-checked with, local data and should be evaluated for decarceral and anti-racist results on a regular basis by the local community, including people impacted by harm and violence, and people impacted by mass incarceration, and their advocates.

 

Principle 3


Pretrial risk assessment instruments must never recommend detention; instead, when a tool does not recommend immediate release, it must recommend a pretrial release hearing that observes rigorous procedural safeguards. Such tools must only be used to significantly increase rates of pretrial release and, where possible, to ascertain and meet the needs of accused persons before trial, in combination with individualized assessments of those persons. Risk assessment instruments must automatically cause or affirmatively recommend release on recognizance in most cases, because the U.S. Constitution guarantees a presumption of innocence for persons accused of crimes and a strong presumption of release pre-trial.

 

Principle 4


Neither pretrial detention nor conditions of supervision should ever be imposed, except through an individualized, adversarial hearing. The hearing must be held promptly to determine whether the accused person presents a substantial and identifiable risk of flight or (in places where such an inquiry is required by law) specific, credible danger to specifically identified individuals in the community. The prosecution must be required to demonstrate these specific circumstances, and the court must find sufficient facts to establish at least clear and convincing evidence of a substantial and identifiable risk of flight or significant danger to the alleged victim (or to others where required by law) before the exceptional step of detention of a presumptively innocent person, or other onerous supervisory conditions can be imposed. All conditions short of detention must be the least restrictive necessary to reasonably achieve the government’s interests of mitigating risks of intentional flight or of a specifically identified, credible danger to others. Any person detained pretrial must have a right to expedited appellate review of the detention decision.

 

Principle 5


Pretrial risk assessment instruments must communicate the likelihood of success upon release in clear, concrete terms. In accordance with basic concepts of fairness, the presumption of innocence, and due process, pretrial risk assessment instruments must frame their predictions in terms of success upon release, not failure. Further, such tools should only predict events during the length of the trial or case – not after the resolution of the open case.

 

Principle 6


Pretrial risk assessment instruments must be transparent, independently validated, and open to challenge by an accused person’s counsel. At minimum, the public, the accused person, and the accused person’s counsel must all be given a meaningful opportunity to inspect how a pretrial risk assessment instrument works. The accused person’s counsel must also be given an opportunity to inspect the specific inputs that were used to calculate their client’s particular categorization or risk score, along with an opportunity to challenge any part – including non-neutral value judgments and data that reflects institutional racism and classism – of that calculation.

Signed by:

  1. African American Ministers In Action
  2. Alternate ROOTS
  3. American-Arab Anti-Discrimination Committee
  4. American Civil Liberties Union
  5. Amistad Law Project
  6. Arab American Institute
  7. Asian Pacific American Labor Alliance
  8. Bend the Arc Jewish Action
  9. Black Alliance for Just Immigration
  10. Black Lives Matters Philadelphia
  11. The Bronx Freedom Fund
  12. Brooklyn Community Bail Fund
  13. Brooklyn Defender Services
  14. The Center for Carceral Communities
  15. Center for Democracy & Technology
  16. Center for Justice Research – Texas Southern University
  17. The Center for Media Justice
  18. Center for Popular Democracy
  19. Center on Race, Inequality, and the Law at NYU Law
  20. Chicago Community Bond Fund
  21. Civil Rights Corps
  22. College and Community Fellowship
  23. Color Of Change
  24. Colorado Freedom Fund
  25. Connecticut Bail Fund
  26. Cville Immigrant Bond Fund
  27. Data & Society
  28. Dauphin County Bail Fund
  29. Decarcerate Tompkins County
  30. Defender Association of Philadelphia
  31. Defending Rights & Dissent
  32. Denver Justice Project
  33. Eastern Iowa Community Bond Project
  34. Electronic Frontier Foundation
  35. Ella Baker Center for Human Rights
  36. Entre Hermanos
  37. Essie Justice Group
  38. Families for Justice as Healing
  39. Fight for the Future
  40. Free Press
  41. FreeThe350BailFund
  42. Global Justice Institute
  43. Government Information Watch
  44. The Greenlining Institute
  45. Helping Educate to Advance the Rights of Deaf Communities (HEARD)
  46. Humanizing AI in Law Research Group, MIT
  47. Immigrant Family Defense Fund
  48. Impact Fund
  49. Impact Justice
  50. Insight Center for Community Economic Development
  51. The Institute of the Black World 21st Century
  52. Jewish Council for Public Affairs
  53. Juntos
  54. Justice Strategies
  55. Kent County (Michigan) Immigrant Bond Relief Fund
  56. LatinoJustice PRLDEF
  57. The Leadership Conference Education Fund
  58. The Leadership Conference on Civil and Human Rights
  59. Local Progress
  60. Madison County Bail Fund Inc.
  61. Massachusetts Bail Fund
  62. The Mass Liberation Campaign
  63. Media Alliance
  64. Media Mobilizing Project
  65. Mijente
  66. Minnesota Freedom Fund
  67. Movement Voter Project
  68. MoveOn
  69. NAACP
  70. NAACP Legal Defense and Educational Fund, Inc.
  71. National Action Network
  72. National Association of Social Workers
  73. National Bail Out
  74. National Center for Lesbian Rights
  75. The National Council for Incarcerated and Formerly Incarcerated Women and Girls
  76. National Council of Churches
  77. National Employment Law Project
  78. National Hispanic Media Coalition
  79. National Law Center on Homelessness & Poverty
  80. NETWORK Lobby for Catholic Social Justice
  81. New America – Public Interest Technology
  82. New America’s Open Technology Institute
  83. Northwest Community Bail Fund
  84. Oakland Privacy
  85. One Pennsylvania
  86. Open MIC (Open Media and Information Companies Initiative)
  87. OVEC-Ohio Valley Environmental Coalition
  88. People’s Action | Mass Liberation Project
  89. People’s Paper Co-op
  90. The People’s Press Project
  91. Philadelphia Bail Fund
  92. Philadelphia Community Bail Fund
  93. Philadelphia Red Umbrella Alliance
  94. PolicyLink
  95. Portland Freedom Fund
  96. POWER Interfaith
  97. Prison Policy Initiative
  98. Progressive Leadership Alliance of Nevada
  99. Project SAFE
  100. Public Defender Association
  101. Public Knowledge
  102. Reclaim Philadelphia
  103. Reentry Think Tank
  104. Richmond Community Bail Fund
  105. Robert F. Kennedy Human Rights
  106. Silicon Valley De-Bug
  107. Southern Center for Human Rights
  108. Southerners On New Ground
  109. Southwest Workers Union
  110. Texas Organizing Project
  111. Tucson Second Chance Community Bail Fund
  112. United Church of Christ, OC Inc.
  113. Urbana-Champaign Independent Media Center
  114. VietLead
  115. Voice of the Experienced
  116. Washington Lawyers’ Committee for Civil Rights and Urban Affairs
  117. Washington Square Legal Services Bail Fund
  118. Young Women’s Freedom Center
  119. 215 People’s Alliance

 

Principle 1


If in use, a pretrial risk assessment instrument must be designed and implemented in ways that reduce and ultimately eliminate unwarranted racial disparities across the criminal justice system.

When considering whether and how to adopt pretrial risk assessment instruments, jurisdictions and stakeholders must acknowledge that the criminal justice system in the United States, since its inception, has allocated benefits and burdens on the basis of race. History confirms the point, as do decades of research and data.

While the methods of arrest, prosecution, and punishment have evolved over time, the imbalance in the allocation of burdens has not. Persistent racial disparities plague the system today. Armed with that knowledge, jurisdictions must work to remedy known, unwarranted racial disparities in the administration of criminal law. In this context, the term “racial disparity” refers to unjustifiable differences in the rates of contact by a racial group with any stage of the criminal justice system that are attributable to explicit bias, implicit bias, socioeconomic inequality, or facially race-neutral policies that produce a disparate racial impact. The national movement to reduce race and wealth-based inequities in pretrial detention will be of limited utility if the types of disparities that jurisdictions seek to remedy simply re-emerge at another point in the system.

Against that historical backdrop, it is imperative that pretrial risk assessment instruments, if used at all, be designed to help meet the goal of reducing racial disparities in the criminal justice system. If a tool cannot help achieve that goal, then it is not a tool that the justice system needs. All pretrial risk assessment instruments must be designed, tested, calibrated, and implemented with the goals of exposing, documenting, reducing, and ultimately eliminating unwarranted racial disparities across the criminal justice system.

Pretrial risk assessment instruments must be designed so that no racial group bears the undue burden of errors made by the instrument. Several measures of fairness – such as error rate balance and predictive parity – currently exist. As innovation progresses, other measures will be developed. Jurisdictions should choose the measure, or combination of measures, that will allow them to best avoid burdening one racial group with erroneous predictions or sanctions. Community input and independent data scientist support in a jurisdiction’s choice of a fairness measure is essential.

Those engaged in the design, implementation, or use of risk assessment instruments should also test ways to reduce the racial disparities that result from using historical criminal justice data, which may reflect a pattern of bias or unfairness. One approach is exploring a corrective weights design, which counteracts biased outcomes by accounting for the effects of systemic racism characterized by patterns of over-policing and other racially disparate criminal justice practices that drive systemic inequities.

The methods chosen to address racial disparities should be jurisdiction-specific, as the causes of disparity may vary over time and by location. Ideally, jurisdictions would use the risk assessment process to engage in research and analysis of the root causes of racial disparities in their criminal justice apparatuses. Critically, the means to address those disparities may fall outside the scope of the criminal justice system, and require an investment in education, public health, economic opportunities, or various other social services.

Jurisdictions should collect and maintain quantitative data on race across all stages of the criminal justice system in order to implement this principle. When possible, that quantitative data should be audited by an agency or institution independent from actors within the system to avoid biased statistical reporting. Quantitative data should also be supplemented by qualitative accounts from individuals who have come into contact with the criminal justice system by virtue of an arrest. These data sets should inform the design, implementation, and use of pretrial risk assessment instruments.

Pretrial risk assessment instruments are not a panacea for racial bias or inequality. Nor are they race-neutral, because all predictive tools and algorithms operate within the framework of institutions, structures, and a society infected by bias. Those facts weigh heavily against their use. However, in those instances when jurisdictions commit to employing a risk assessment instrument as part of the pretrial ecosystem, doing so with an eye toward eliminating unwarranted racial disparities advances the cause of justice for all.

 

Principle 2


If in use, a pretrial risk assessment instrument must never recommend detention; instead, when a tool does not recommend immediate release, it must recommend a pretrial release hearing that observes rigorous procedural safeguards. Such tools must only be used to significantly increase rates of pretrial release and, where possible, to ascertain and meet the needs of accused persons before trial, in combination with individualized assessments of those persons. Pretrial risk assessment instruments, when used to assign community supervision, should only be used to assign the least restrictive and onerous forms of pretrial supervision. Risk assessment instruments must automatically cause or affirmatively recommend release on recognizance in most cases, because the U.S. Constitution guarantees a presumption of innocence for persons accused of crimes and a strong presumption of release pretrial.

Pretrial risk assessment instruments, if used at all, should only identify groups of people to be released immediately. Additionally, such tools may be used in combination with an individualized assessment to determine the specific needs of an accused person pretrial, with the goal of finding non-punitive, non-onerous direct supports and alternatives to detention that would encourage the accused person’s appearance at trial, and would reduce their risk of being arrested again during the length of the trial.

Pretrial risk assessment instruments must be carefully crafted to avoid encroaching upon the limited scope, or prohibition, of preventative detention and over-conditioning of release in each jurisdiction. Given these limits, pretrial risk assessment instruments need not forecast future arrests at all. Such an approach would alleviate the most egregious forms of bias, and acknowledge the fact that bail has historically existed only for the purpose of ensuring appearance at court. Additionally, the design of any tool should always give far greater weight to the avoidance of false positives than to the avoidance of false negatives, as the harms of detaining the wrong person pretrial are far greater than erring on the side of release.

If jurisdictions seek to use such tools, any scores that do not support or recommend release needn’t be shown to individual judges. Rather, as discussed in the preamble, they need only trigger individual hearings. To the extent that tools involve risk scores being given to individual judges for decisionmaking and value-balancing, the results of pretrial risk assessment instruments should never be framed in a way that recommends detention or that labels an individual as “high risk,” as this masks an inherent value-balancing in the guise of purportedly neutral data. An algorithm cannot determine what level of risk warrants detention or release. Such determinations are fundamentally political and not empirically derived, and they must be explicitly identified as such. Moreover, unquestioning reliance on risk assessment instruments – when determining a defendant’s pretrial liberty – directly violates the constitutional requirement of individualized determinations.

Data used in the development of risk assessment instruments must be reviewed for accuracy and reliability. Data collection must include a transparent and periodic examination of release rates, release conditions, technical violations or revocations, and performance outcomes by race to monitor for disparate impact within the system.

Defense counsel must be included in the process of selecting, calibrating, designing, shaping, and testing a pretrial risk assessment instrument and included in the ongoing evaluation of the tool. Tools of this nature should be overseen and used only toward reducing or ending monetary bail without triggering more expansive forms of supervision and surveillance, such as the use of electronic monitoring. Instead, such controls should only be imposed as an actual alternative to incarceration for someone who has otherwise been lawfully determined eligible for possible detention. Groups with a stake in increasing detention, or in causing high money bail amounts to be set, should not be involved in tool selection.

If the use of a particular pretrial risk assessment instrument by itself does not result in an independently audited, measurable decrease in the number of people detained pretrial, the tool should be pulled from use until it is recalibrated to cause demonstrably decarceral results.

 

Principle 3


Neither pretrial detention nor conditions of supervision should ever be imposed, except through an individualized, adversarial hearing. The hearing must be held promptly to determine whether the accused person presents a substantial and identifiable risk of flight or (in places where such an inquiry is required by law) specific, credible danger to specifically identified individuals in the community.

The prosecution must be required to demonstrate these specific circumstances, and the court must find sufficient facts to establish at least clear and convincing evidence of a substantial and identifiable risk of flight or significant danger to the alleged victim (or to others where required by law) before the exceptional step of detention of a presumptively innocent person, or other supervisory conditions can be imposed.

All conditions short of detention must be the least restrictive necessary to reasonably achieve the government’s interests of mitigating risks of intentional flight or of a specifically identified, credible danger to others.

Any person detained pretrial must have a right to expedited appellate review of the detention decision.

Extensive evidence demonstrates that pretrial detention and intensive supervision can and do derail the lives of accused people, imposing enormous costs on them, their loved ones, and society at large. Detention and intensive supervision are extraordinary and toxic sanctions to impose on any person based on an unproven allegation of criminal conduct. These deprivations cannot be justified by merely resorting to generalities or statistical estimates. Only an individual, adversarial hearing should ever be considered a constitutionally sound basis for detention pretrial.

At timely, individualized hearings, counsel must be provided to represent accused persons. Public defenders or court-appointed counsel shall be present to represent those who cannot afford their own attorney. The court must ensure that defense counsel has the time, training, and resources to learn important information about the client’s circumstances that may not be captured in a pretrial risk assessment instrument and adequate opportunity to present that information to the court. The court shall administer an “ability to pay” inquiry prior to setting any financial conditions of release, and only after concluding financial conditions are least restrictive. When an accused person establishes that they receive means-tested public benefits, currently held or recently held in correctional facilities or residential facilities that serve mental or physical illness, or have an income below 250 percent of the federal poverty guidelines, when adjusting for household size, said individuals shall be determined ineligible to pay. When an accused person suggests that they do not meet the above-mentioned criteria, but otherwise believe they are unable to pay the amount established by the court, said individuals shall receive a substantive ability-to-pay hearing, including inquiry into income, debt, and public benefits, as well as an affirmative opportunity for accused individuals to provide evidence of indigence. Prospective future employment, access to illiquid assets, the ability to borrow money from family members, or the ability to loan money from a for-profit surety company shall not be considered as contributing to an accused individual’s present ability to pay.

In all cases, the hearing shall commence with a presumption of release on recognizance. If the prosecution seeks an order of pretrial detention – which must be done transparently, and not through the imposition of unaffordable money bail – the burden of proof shall be on the prosecution to establish fact-specific clear and convincing evidence of “relevant risks,” defined as a substantial and identifiable risk of intentional flight or, when explicitly permitted by law, a specifically identified, credible danger to a specific person (rather than the amorphous notion of “community safety”).

Detention should only be an option in specific and narrow circumstances; in all other cases, only release on recognizance and, where necessary, non-monetary, non-onerous conditions of release, should be available. In the specific and narrow circumstances where pretrial detention is a possibility, this determination may only be made at the conclusion of an individualized, adversarial hearing following the appropriate procedures, and if the court finds at least clear and convincing evidence of a relevant risk that no condition or combination of conditions of release or support services could mitigate.

Prior to setting any nonfinancial conditions on an arrestee’s pretrial release, the court must determine that those conditions are the least restrictive necessary to reasonably assure that any relevant needs are served or risks present are mitigated. In all cases, the presumption will be of release on no conditions. Any conditions set must be tailored to the individual arrestee: uniform or blanket conditions of release shall not be permitted.

 

Principle 4


If in use, a pretrial risk assessment instrument must be transparent, independently validated, and open to challenge by an accused person’s counsel. The design and structure of such tools must be transparent and accessible to the public.

The design of pretrial risk assessment instruments must be public, not secret or proprietary. This means adopting local legislation or enforceable regulations that enforce transparency by sharing the data and design with that specific jurisdiction, in addition to reporting requirements throughout the implementation process. The source code and training data (appropriately anonymized) should be made public. And all tools and their documentation must be clear about the source data and code underlying each conclusion in any report; in other words, whether any given conclusion is empirically derived or based on a political, moral or personal choice or assumption set by criminal justice decisionmakers or other government officers.

At minimum, the public, the accused person, and the accused person’s counsel must all be given a meaningful opportunity to inspect how a pretrial risk assessment instrument works, including access to:

  • A complete description of the design and testing process used to create a tool;
  • All input factors a pretrial risk assessment instrument uses;
  • The weights assigned to those factors;
  • The thresholds and specific data used to determine “low,” “medium,” or “high” risk calculations or risk scores; and
  • The outcome data used to develop and validate the tool.

The accused person’s counsel must also be given an opportunity to inspect the specific inputs that were used to calculate their client’s particular categorization or risk score, along with an opportunity to challenge any part – including non-neutral value judgments and data that reflects institutional racism and classism – of that calculation.

In addition, the factors considered by a pretrial risk assessment instrument must comport with local and state law regarding the presumption of release.

 

Principle 5


If in use, a pretrial risk assessment instrument must communicate the likelihood of success upon release in clear, concrete terms.

In accordance with basic concepts of fairness, the presumption of innocence, and due process, pretrial risk assessment instruments must frame their predictions in terms of success upon release, not failure. For example, a tool could say that, in the past, people similar to the accused person have had an 80 percent likelihood of appearing for all of their court dates over 18 months of hearings. Communicating and framing a pretrial risk assessment instrument’s prediction in negative terms (that is, the likelihood the accused would fail to appear upon release) may unnecessarily lead decisionmakers to perceive and treat the accused more harshly. To the extent possible, the tools should also provide the likelihood of success when certain supports or services are provided, consistent with identifying and addressing genuine needs rather than purported risks.

Further, such tools should only predict events during the length of the trial or case – not after the resolution of the open case. Every tool should thus have a temporal cutoff for its prediction period – for example, six months – which local stakeholders may wish to track speedy trial laws.

 

Principle 6


If in use, a pretrial risk assessment instrument must be developed with community input, revalidated regularly by independent data scientists with that input in mind, and subjected to regular, meaningful oversight by the community.

A pretrial risk assessment instrument must be designed with input from members of the local community, including, but not limited to:

  • Independent data scientists not paid for by the designer of the implemented tool;
  • Survivors of harm and violence and people impacted by mass incarceration;
  • Decisionmakers and actors throughout the criminal justice system, including public defenders, judges, and district attorneys; and
  • Community groups focused on racial justice.

Revalidation of the tool should include a check for predictive validity – including understanding mean score differences across race/gender/protected characteristics that could result in disparate impact on those communities. Revalidation should include independent auditing for the decarceral and racially equitable results our communities want and need. Validation and revalidation should take place annually at the least, with the ideal being quarterly.

The people described above should be convened into a funded and staffed community advisory board, supported by data scientists to understand how algorithms work, how they are trained, how they weight their factors, and how pretrial decisionmakers use the algorithms in real decisions. This board should be able to host regular hearings educating it and the public on how algorithms are used in pretrial decisionmaking, and potentially in other criminal justice, community well-being, or public sector decisionmaking contexts. This funded board could be convened through acts of city councils, mayors, or state bodies, depending on local laws and jurisdictional practices. This funded board should have the ability, in concert with elected officials, to pause or roll back the use of a pretrial risk assessment instrument if that tool is not meeting, in practice, the decarceral or racially equitable goals set out by the funded board and other stakeholders.

Judges and their officers – including bail administrators and magistrates, pretrial services staff, and anyone who touches the pretrial risk assessment instrument – should receive regular training and retraining on the use of the tool, and should be overseen in their use of the tool for decarceral and racially equitable results.

The particular pretrial risk assessment instrument designed or chosen should be trained by, or at least cross-checked with, local data and should be evaluated for decarceral and racially equitable results on a regular basis by the local community, including the funded board consisting of key groups mentioned above.

Splash Statement