Validating AI Jury Selection Tools: A Blind Mock-Panel Test for Accuracy Claims

If a vendor tells you its AI picks favorable jurors at a rate attorneys can't match, don't take the pitch deck's word for it. Run a blind mock-panel test: preregister what the tool is supposed to predict, compare its flags against independent attorney judgments on a holdout panel nobody tuned the model against, break out the error rates by subgroup, and require the vendor to hand over confidence intervals and an audit log before you sign anything. If a vendor won't support that test, that's your answer.

This isn't a new idea borrowed from marketing science. It's the same basic approach courts already use to vet expert methodology. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), asks whether a method has been tested, has a known error rate, has been peer reviewed, and is generally accepted.[1] If that's the bar for an expert witness's methodology at trial, it's a reasonable bar for the methodology you're about to pay a vendor to run on your venire.

StrikeList has already covered what AI jury tools are and how to shop for one in general terms. This post is narrower. It's a protocol you can actually run, or hand to an associate to run, before you commit budget to a tool based on a headline accuracy number.

Why AI jury selection claims need testing, not trust

Vendors in this space tend to advertise a single number: 80 percent accurate, outperforms consultants, predicts strikes better than instinct. The number is almost never accompanied by a sample size, a holdout set, a confusion matrix, or a subgroup breakdown. That's a problem because a single impressive demonstration case tells you almost nothing about how the tool performs on your venire, in your venue, on your case type.

One published preprint illustrates both the promise and the limit of a single-case result. In a mock wrongful-termination case, a model reportedly reached accuracy of 0.818 and 0.796 across two configurations, compared with 0.693 for the majority vote of professional jury consultants.[5] That's a genuinely interesting data point. It is not proof the tool generalizes to a criminal voir dire in a different county with a different jury pool and a different claim type. One case, one fact pattern, one jury pool. Treat it as a hypothesis worth testing again, not a conclusion you can rely on.

','data':[0.818,0.796,0.693],'backgroundColor':['%234e79a7','%23f28e2b','%23e15759']}]},'options':{'scales':{'yAxes':[{'scaleLabel':{'display':true,'labelString':'Accuracy%20(proportion%20correct,%20one%20mock%20trial)'},'ticks':{'max':1}}]}}} )

Figure: results from a single reported mock wrongful-termination trial; not a generalizable benchmark across venues or case types.[5]

The legal standard behind the protocol

Federal Rule of Evidence 702 requires sufficient facts or data, reliable principles and methods, and reliable application of those methods to the facts of the case.[9] Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999), extended that reliability inquiry beyond traditional scientific testimony to any technical or specialized method, which is exactly the category an AI jury-selection tool falls into.[6] If an AI-informed strike decision later becomes relevant to a Batson challenge or a motion attacking your selection process, you want to be the lawyer who can point to a documented, reproducible test. Not the one explaining, for the first time, under questioning, that you never asked the vendor for an error rate.

A blind mock-panel validation protocol

Here's the protocol broken into steps you can actually execute, either in-house with a retained jury consultant or as a condition you impose on the vendor before purchase.

Step 1: Define the claim before you test anything

Vendors say "more accurate." That's not a testable claim. Force it into something measurable: does the tool identify jurors who favor one side, predict which jurors a given side will strike, rank cause-for-cause risk, or flag issue-specific bias (say, bias against corporate defendants in a products case)? Specify the venue, the case type, the prediction horizon, and the decision threshold the tool uses to flag a juror. If the vendor can't state the claim this specifically, you can't test it, and neither can they.

Step 2: Preregister the design before anyone sees AI output

Write down the hypothesis, target sample size, labels, exclusion criteria, scoring rules, subgroup definitions, and statistical tests before the AI scores exist. This matters because it prevents the most common failure mode in vendor demos: tuning the narrative to whatever the model happened to get right. Use a holdout set the vendor never saw and can't adjust against. This is the step that separates a real validation from a sales demo dressed up as a pilot.

Step 3: Recruit and blind a mock panel

Pull mock jurors who resemble the actual venire on demographics, geography, and case-relevant experience, not a convenience sample of law students or office staff. Give every participant identical case materials and a standardized questionnaire. Critically: the attorneys making comparison judgments should never see the AI's scores, and the AI should only receive the inputs it would actually get in real deployment, no extra data, no cleaned-up transcripts. Randomize presentation order. Keep the raw responses.

Step 4: Build an independent human comparator

Have several experienced trial attorneys independently rank or classify each mock juror using the identical information the vendor's tool received. Don't let them confer until after scoring. Reconcile disagreements only under a rule you wrote down in advance, not in the moment. Measure inter-rater agreement among your attorneys first. If your own attorneys disagree with each other 30 percent of the time, an AI "error" against any one attorney's judgment may just be reasonable professional disagreement, not a model failure.

Step 5: Score more than one overall accuracy number

A single accuracy figure hides more than it reveals. Require sensitivity, specificity, positive and negative predictive value, a full confusion matrix, and calibration metrics. For continuous risk scores, require a calibration plot and a Brier score. And set the bar for "meaningful improvement" before you see results: a statistically significant difference of two points over attorney judgment may not be worth the cost and liability exposure of adopting the tool.

Why auditing subgroup error rates in AI jury selection is non-negotiable

This is the step vendors are least likely to volunteer and the one that matters most if the tool's output ever gets scrutinized in a Batson context or an ethics inquiry.

Report false-positive and false-negative rates broken out by race, sex, age, disability status, language, socioeconomic proxy, and case-relevant experience, to the extent doing so is legally and ethically permissible. Compare calibration and selection rates across those groups. A model that looks accurate overall can still be systematically overflagging one group as high-risk, and overall accuracy will not reveal that; only the subgroup breakdown will.

Bar association guidance in this space is consistent on one point even where it doesn't bless or prohibit specific tools outright: lawyers should not rank or score jurors by protected characteristics or close proxies for them, and any AI output needs to be independently verified before it informs a strike decision.[7] An unaudited subgroup error rate is exactly the kind of gap that turns a convenient tool into a discoverable liability.

Different stakeholders, different stakes on subgroup error

A plaintiff's attorney relying on an AI flag to strike a juror who skews toward corporate defendants needs to know if that flag correlates with a protected characteristic the model is using as a silent proxy. A criminal defense attorney striking jurors perceived as pro-prosecution faces the same exposure, amplified by the higher stakes of a Batson challenge in a criminal case where the record gets scrutinized on appeal. Corporate defense counsel evaluating a vendor for a large products or employment docket should be asking whether the tool's error rates differ across the demographic makeup of the specific venues they try cases in, not just a national or vendor-curated sample. And prosecutors, who face their own Batson exposure under Batson v. Kentucky, 476 U.S. 79 (1986), and its progeny, have the same reason as anyone to want a documented, reproducible error rate rather than an unverified vendor claim sitting in a file somewhere.

Step 6: Require uncertainty, reproducibility, and audit logs

No validation is complete without a measure of how confident you should be in the numbers. Require bootstrap or binomial confidence intervals around every principal metric and every subgroup result. Require disclosure of how the vendor handles missing data and whether the model or its version changed between your test and your eventual deployment. Rerun the test on the same locked inputs and confirm you get the same output; a result that can't be reproduced from identical inputs is the kind of thing that falls apart under a Rule 702 reliability challenge.[9][10]

Just as important: require an exportable audit log for every scored juror, showing the input fields, any transformations applied, the model version, the prompts used, the raw output, a timestamp, the user, and any subsequent human override. If you can't reconstruct, after the fact, exactly what the tool saw and said about a specific juror, you have no way to defend that decision if it's ever challenged.

Procurement rule: don't buy on a headline number

Before any contract gets signed, require independent replication on a second mock panel you control, written disclosure of training-data provenance and exclusions, subgroup results with confidence limits, clear data retention and deletion terms, and contractual access to the audit logs described above. Treat a vendor's refusal to support blinded testing, or an unwillingness to disclose methodology and error estimates, as a failed validation. It is not evidence of proprietary superiority. It is evidence you can't test the claim, which means you can't rely on it.

FAQ

What does "blind" mean in a blind mock-panel test of an AI jury tool? It means the attorneys scoring the mock jurors never see the AI's output while making their own judgments, and the AI only receives the same inputs it would get in real deployment. This prevents either side's judgment from being influenced by the other's conclusions, which is what makes the comparison meaningful.

Is a single vendor case study, like the 0.818 accuracy figure from one mock trial, enough to justify adoption? No. A single reported result from one mock case, such as the 0.818 and 0.796 accuracy figures reported against a 0.693 jury-consultant baseline in one wrongful-termination scenario, is a data point worth noting, not a generalizable benchmark.[5] It doesn't account for different venues, case types, or jury pools, which is exactly why a repeatable protocol on your own venire matters more than any vendor's published demo.

What legal standard should attorneys use as the benchmark for testing these tools? The Daubert factors, tested methodology, known error rate, peer review, and general acceptance, are a practical framework even outside the courtroom.[1] Federal Rule of Evidence 702 and Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999), extend that reliability scrutiny to any technical method, which includes AI jury-selection tools if their output ever becomes relevant to a challenge.[6][9]

Why does subgroup error rate matter if the tool's overall accuracy looks good? Overall accuracy can mask systematic overflagging of one demographic group even when the aggregate number looks strong. Bar guidance cautions against ranking jurors by protected characteristics or close proxies and stresses independent verification of AI output before it's used.[7] A subgroup audit is the only way to catch that kind of bias before it becomes a Batson problem.

Who should run this validation, the law firm or the vendor? Both, ideally. The vendor should be contractually required to support blinded testing, disclose methodology, and hand over confidence intervals and audit logs. But the firm or a retained jury consultant should control the holdout panel and the preregistered design, since a vendor grading its own homework defeats the purpose of the test.

Closing

None of this requires a data science department. It requires writing down what you're testing before you test it, keeping the humans and the AI blind to each other's work until scoring is done, and refusing to accept a headline number without a subgroup breakdown and an audit trail behind it. That's the same discipline trial lawyers already apply to expert testimony; there's no reason to waive it just because the "expert" in this case is a vendor's model.

StrikeList AI was built with this kind of scrutiny in mind, with documented methodology and output attorneys can actually trace back to inputs. Whatever tool you're evaluating, including this one, the protocol above is a reasonable bar to hold it to before it touches your voir dire.

Notes

1. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).
2. Batson v. Kentucky, 476 U.S. 79 (1986).
3. Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999).
4. Fed. R. Evid. 702.
5. Preprint reporting mock wrongful-termination trial accuracy comparison (AI model: 0.818 and 0.796; jury consultant majority vote: 0.693), as cited in source research; preprint title and URL not independently verified for this post.