Legal research with LLMs: the prompts and harnesses
Felix Ringe and an anonymous co-author describe using LLMs to find possible defects in German regulations. Their examples concern the relationship between a regulation and the statutory authority cited for it. That raises an engineering question: what prompts and evaluation machinery can make this kind of research reproducible?
I asked for a survey of the public research methods: prompts, datasets, harnesses, and verification. The useful material falls into several distinct designs. There is public code to start from. Ringe’s article does not disclose its discovery setup.
Read the full research report for exact prompt locations and evaluation limits, or start with the source table below.
What the public methods offer
SocioHack studies reward optimization in simulated institutions. It releases environment data, training code, and model-based simulation and scoring. Loophole offers a simpler prototype: separate roles find gaps and overreach, propose revisions, and review them against earlier cases.
Legal Zero-Days uses expert-created defects in modified legislation, with an Inspect evaluation. Its public appendix redacts the main prompt. The ambiguity study supplies a particularly useful control: test whether a model understands the intended meaning separately from the interpretation it chooses.
These designs answer different questions. A high simulator reward, a detected planted defect, and a supported finding about existing law are different results. The source table below links directly to material worth reading first.
Sources to start with
| Source | Starting point |
|---|---|
| Ringe’s article | The motivating work. Its discovery prompts and harness are not disclosed. |
| SocioHack | Environment data, simulator and scoring prompts, RL training, and evaluation. |
| Loophole | Explicit finder, overreach, judge, drafting, and revision-review prompts. |
| Legal Zero-Days | Expert-planted legislative puzzles and Inspect code. Main prompt redacted. |
| Ambiguity study | Prompt generators, intent probes, unambiguous controls, and recorded results. |
| LegalWiz | Methods for generating and detecting legal-document contradictions. |
| Replication workflow | Harbor tasks and verifiers for a separate legal research project. |
| Inspect | A reusable evaluation harness with scorers and logs. |
| LegalBench / LegalBench-RAG | Retrieval benchmark and legal reasoning tasks for prerequisite checks. |
The hard part is verification
For an existing legal system, the missing step is often evidence outside the prompt: dated legislation, amendments, alternative authority, and judicial interpretation. A second model agreeing with a claim does not settle those questions.
A useful harness should preserve exact source spans, seek contrary evidence, include clean controls, and keep proposed findings separate from reviewed findings. My longer report maps the prompt files, compares the evaluation designs, and proposes a workflow with those checks. It includes a fuller source table and fixed source-code snapshots.
Evidence reviewed: publications and selected source code, 31 October 2026. No model experiments were run for this survey.