PromptLock

CI for prompts. Fail the pull request when a prompt, model, or parameter change silently makes your LLM outputs worse.

Your test suite catches broken code. Nothing catches a prompt edit that quietly stops returning parseable JSON on half your inputs. Detecting that output changed is trivial — LLMs are stochastic, so any tool that diffs strings fires on whitespace and gets muted in a week. Detecting that it got worse is the product.

Try it here

Explore all 19 changes → Pick any prompt or model change and see what PromptLock decides, and why A failing report → Sonnet downgraded to Haiku: 6 of 50 regressed, 73% cheaper A clean report → All 50 cases hold — what a quiet PR looks like

Everything here is real output from promptlock check, generated by scripts/build_site.py. Nothing is mocked up.

Does it work?

6 prompt edits that genuinely degrade output, 10 a reviewer would wave through, 3 model/parameter changes. 50 cases each.

Detector	Recall	False positives	Precision
Naive string diff	6/6	10/10	38%
PromptLock	6/6	0/10	100%

Both find every real regression. Only one is quiet enough to leave switched on. That gap is the entire product.

Run it yourself

git clone https://github.com/tan8696/promptlock
cd promptlock && pip install -e .

promptlock record     # snapshot known-good behaviour
promptlock check      # 50 pass · 0 drift · 0 fail
python3 scripts/benchmark.py

No API key, no network, no account — the demo runs against a behavioural mock, so the numbers above reproduce exactly.