Prompts and harnesses for legal defect research: report
This report surveys public prompts and evaluation harnesses for research into defects in legal rules. A harness is the software that supplies inputs, runs a model, records outputs, and evaluates results.
The starting point is Felix Ringe’s article on possible defects in German regulations. The survey follows the research methods, rather than collecting actionable examples. Evidence reviewed on : publications, project documentation, and selected source files. No model experiment or legal claim was independently reproduced.
Scope and evidence
Ringe and an anonymous co-author describe using LLMs to identify regulations whose provisions may lack a required citation of statutory authority. Their article supplies legal arguments and objections. It does not disclose discovery prompts, model versions, an automated search procedure, or a runnable harness. This survey found no public implementation linked from that article. That finding does not establish that none exists elsewhere. [S1]
Three activities recur across the literature: detecting a planted defect, searching a simulated rule system, and investigating a candidate defect in existing law. They have different success criteria. A simulator score does not establish legal validity. An expert-written puzzle supplies an expected answer that open-ended legal research usually lacks.
Searches covered the author’s pages, papers cited there, regulatory loophole benchmarks, ambiguity experiments, legal contradiction detection, and public code repositories. Search engines supplied leads. The findings below rely on the linked primary material. Selected code was read directly, without installation. Release completeness was checked only to the extent stated for each project.
Source table: a starting point
The table separates research papers from executable artifacts. Source links use paper versions or fixed Git commits where this review inspected a specific version.
| ID | Source | What to use it for |
|---|---|---|
| S1 | Ringe: Legal exploits may emerge at scale | Motivation and legal arguments. Discovery prompts and harness were not published on the page reviewed. |
| S2 | Large Language Models Hack Rewards, and Society | Environment design, RL and non-RL methods, metrics, and human checks. See appendices B–D. |
| S3 | SocioHack: source snapshot | Training and evaluation implementation. Start with src/reward.py and eval/eval_historical.py. |
| S4 | Loophole: prompt definitions | Finder, overreach, drafting, judgment, and regression-review prompts. Read with the judge implementation. |
| S5 | Legal Zero-Days: paper and appendix code | Expert-created legislative puzzles and an Inspect evaluation. Main system prompt is redacted. |
| S6 | Language Models Identify Ambiguities and Exploit Loopholes | Controlled ambiguity experiments, intent probes, and unambiguous controls. |
| S7 | Ambiguity study: prompt generator | Exact generated prompts and experimental variants. Repository also contains recorded outputs. |
| S8 | LegalWiz: paper | Synthetic document contradictions, hybrid detection, and human annotation. Full runnable release not confirmed in this review. |
| S9 | Ringe: Reproduce before publishing | Harbor tasks, verifiers, and trajectories for a separate legal research project. |
| S10 | Inspect documentation | Evaluation framework: datasets, solvers, tools, scorers, and logs. |
| S11 | LegalBench | Task-specific legal reasoning prompts and datasets. Useful prerequisite checks. |
| S12 | LegalBench-RAG | Retrieval evaluation against relevant source spans. Does not measure novel defect discovery. |
What each approach measures
| Approach | Input and evaluation | Main limit |
|---|---|---|
| Ringe | Existing regulations, enabling statutes, and legal argument | Discovery procedure not disclosed |
| SocioHack | Simulated institutions, outcome rewards, and held-out patches | Results depend on simulator and judge behavior |
| Loophole | Stated principles, generated rules, candidate cases, and model review | Review does not prove legal validity or consistency |
| Legal Zero-Days | Legislation with expert-planted defects and expert target answers | Main prompt redacted and puzzle release not located |
| Ambiguity study | Controlled instructions, numerical choices, and intent controls | Small language tasks rather than a legal corpus |
| LegalWiz | Synthetic contradictions with annotated sentence pairs | Contradiction detection covers only part of legal defect research |
SocioHack: optimization inside a simulated institution
The paper introduces 72 environments: 32 historical, 20 synthetic, and 20 fictional. Its policy generates strategies under written rules. A simulator predicts outcomes, a scorer assigns reward, and generated constraints change subsequent search. Held-out patches supply reference vulnerabilities. The paper also discusses direct prompting and non-RL search baselines. [S2]
The public repository exposes GRPO training, environment data, simulator and scoring prompts, constraint generation, and evaluation scripts. This review did not establish that every baseline and analysis in the paper has a complete executable release. [S3]
| Location | Role |
|---|---|
| DEFAULT_SIMULATOR_PROMPT | Convert a proposed strategy into events and state variables. |
| DEFAULT_JUDGE_PROMPT | Calculate points from the rubric and simulated state. |
| generate_constraint() | Describe a rule gap and generate a further constraint. |
| eval/eval_historical.py | Judge whether a mined strategy targets the vulnerability addressed by a reference patch. |
The constraint prompt asks for rules that can cap further gains rather than simply ban the behavior. That choice shapes exploration. It should remain explicit in any adaptation.
The paper’s human checks report moderate agreement with its matching judge. They also distinguish a feasible plan from a genuine loophole. Both points matter when interpreting an automated discovery count. [S2]
Loophole: explicit roles and accumulated cases
Loophole starts with stated principles and drafts rules. A finder proposes cases the rules permit despite conflicting with those principles. A second finder looks for overreach. A judge proposes revisions or escalates conflicts, and a separate prompt checks revisions against prior decisions. [S4]
| Prompt | Research function |
|---|---|
LOOPHOLE_FINDER_SYSTEM | Generate concrete cases, including definition boundaries, exceptions, and interactions between permitted actions. |
OVERREACH_FINDER_SYSTEM | Find prohibited cases that the principles would permit. |
JUDGE_RESOLVE | Propose a minimal correction or identify a conflict. |
JUDGE_VALIDATE | Review a correction against resolved cases. |
LEGISLATOR_REVISE | Apply the resolution while retaining prior decisions. |
The judge implementation confirms that validation is another model call. Accumulated cases are useful regression inputs. Passing this review is neither a proof of consistency nor a finding about actual law. Treat the repository as a prototype whose prompts can be studied independently of its runtime.
Legal Zero-Days: expert-created legislative puzzles
Sadler and Sherburn use lawyers to construct puzzles from original legislation, a modified instrument, and an expert explanation. The model reviews the legislation for consequential operational failures. A model grader compares the response with the expert target. Internet search was excluded from the evaluation. [S5]
Appendix B shows an Inspect task: load a JSON dataset, add a system message, generate a response, score it, and repeat trials. Appendix A shows judge-validation code. The system message is redacted. This review did not locate a public puzzle download, so the appendix alone cannot reproduce the experiment.
This design is useful for a known-answer evaluation. It measures detection of planted defects under bounded context. Extending it to existing law requires additional evidence retrieval and a process for adjudicating unexpected answers.
Ambiguity experiments: distinguish understanding from behavior
The EMNLP 2025 ambiguity study tests scalar implicature, structural ambiguity, and scenarios with power differences. It includes unambiguous controls and separate questions about intended meaning. These help distinguish a favorable interpretation from failure to understand the instruction. [S6]
The prompt generator exposes the experimental variants directly. The repository also contains scenario prompts, model wrappers, evaluation code, and recorded outputs. These are useful controls for a new harness, although the tasks do not test large-scale statutory research. [S7]
Contradiction detection, retrieval, and replication
LegalWiz generates legal documents with controlled contradictions. It combines natural-language inference, an LLM judge, and human annotation. The paper identifies prompt appendices for generation, contradiction injection, and judging. This review inspected the published method but did not confirm a complete runnable release or transcribe those appendix prompts. [S8]
LegalBench supplies task-specific reasoning prompts and datasets. LegalBench-RAG evaluates retrieval against relevant source spans. They can expose prerequisite failures before an open-ended search begins. Neither benchmark directly measures novel legal defect discovery. [S11] [S12]
Ringe’s separate reproduction article links Harbor tasks, verifiers, and agent trajectories for interpretive-canons research. It describes giving an independent agent methods and inputs, then checking reconstructed outputs against withheld references. It does not document the legal-exploits discovery setup. [S9]
Inspect provides reusable datasets, solvers, scorers, tools, and logs. Its direct use in Legal Zero-Days makes it a relevant framework to inspect first. A framework supplies execution and records. Researchers still need to define what counts as a legally supported result. [S10]
A proposed harness for evidence-based legal review
The following design is this report’s synthesis. It has not been implemented or evaluated. It combines explicit roles, experimental controls, source retrieval, and independent review.
- Freeze the evidence. Record jurisdiction, research date, document versions, commencement dates, amendment instruments, and retrieved source text.
- Separate interpretation from defect generation. Ask for an ordinary reading before asking for candidate inconsistencies. Include clean documents and planted defects as controls.
- Require a compact candidate record. Capture the exact provisions, source spans, claimed mismatch, necessary assumptions, and a falsifiable legal proposition.
- Search for disproof. Retrieve contrary authority, alternative enabling powers, predecessor provisions, transitional rules, and relevant judicial interpretation.
- Review independently. Give a reviewer the evidence and claim. Record supported, rejected, and unresolved outcomes, with reasons. Require qualified legal review before asserting legal validity.
- Test any proposed correction. Repeat both the original case and legitimate neighboring cases. A correction that blocks everything has not preserved the intended rule.
Report retrieval quality, citation accuracy, false positives on clean controls, confirmed detections, unresolved candidates, cost, and run-to-run variation separately. Keep the candidate generator’s output distinct from the reviewer’s verdict. Count duplicate mechanisms once under a documented rule.
A useful candidate record could contain corpus_version, provisions, supporting_spans, contrary_spans, assumptions, claim, review_status, and review_reason. These fields are a proposed research format, not a schema taken from the surveyed projects.
Where to start
Read Loophole’s prompt definitions for the clearest separation of roles. Read the ambiguity study for controls, and Legal Zero-Days for an expert-target evaluation design. Use SocioHack when the research question concerns iterative optimization inside a simulator. Inspect and the retrieval benchmarks supply supporting infrastructure.
The central unresolved issue is verification against the legal system outside the prompt: primary authority, dates, amendments, and judicial interpretation. A plausible candidate becomes a research result only after that evidence supports the claim.
Review boundary: selected source inspection only. No dependency installation, model run, benchmark reproduction, or independent determination of legal validity formed part of this survey.