PromptLock — model downgrade

6 of 50 cases regressed.

pass
43
drift
1
fail
6
median drift
0.0357 / 0.025 (systemic)
judge calls
12
total cost
$0.02363

What changed

0.0000.0380.0770.1150.154t001: drift 0.0371, threshold 0.12 (FAIL)t002: drift 0.0218, threshold 0.12 (PASS)t003: drift 0.0339, threshold 0.12 (PASS)t004: drift 0.04, threshold 0.12 (PASS)t005: drift 0.0171, threshold 0.12 (PASS)t006: drift 0.0287, threshold 0.12 (PASS)t007: drift 0.022, threshold 0.12 (PASS)t008: drift 0.0195, threshold 0.12 (PASS)t009: drift 0.0139, threshold 0.12 (PASS)t010: drift 0.0185, threshold 0.12 (PASS)t011: drift 0.0351, threshold 0.12 (PASS)t012: drift 0.0325, threshold 0.12 (PASS)t013: drift 0.0256, threshold 0.12 (PASS)t014: drift 0.0176, threshold 0.12 (PASS)t015: drift 0.0193, threshold 0.12 (PASS)t016: drift 0.0859, threshold 0.12 (PASS)t017: drift 0.0618, threshold 0.12 (PASS)t018: drift 0.0852, threshold 0.12 (PASS)t019: drift 0.0697, threshold 0.12 (PASS)t020: drift 0.0997, threshold 0.12 (PASS)t021: drift 0.0067, threshold 0.12 (PASS)t022: drift 0.0363, threshold 0.12 (PASS)t023: drift 0.1044, threshold 0.12 (FAIL)t024: drift 0.0086, threshold 0.12 (PASS)t025: drift 0.0247, threshold 0.12 (PASS)t026: drift 0.0081, threshold 0.12 (PASS)t027: drift 0.0779, threshold 0.12 (PASS)t028: drift 0.0249, threshold 0.12 (PASS)t029: drift 0.0184, threshold 0.12 (PASS)t030: drift 0.039, threshold 0.12 (PASS)t031: drift 0.0317, threshold 0.12 (PASS)t032: drift 0.0186, threshold 0.12 (PASS)t033: drift 0.0363, threshold 0.12 (FAIL)t034: drift 0.0939, threshold 0.12 (PASS)t035: drift 0.1336, threshold 0.12 (DRIFT)t036: drift 0.0336, threshold 0.12 (PASS)t037: drift 0.11, threshold 0.12 (FAIL)t038: drift 0.086, threshold 0.12 (PASS)t039: drift 0.1108, threshold 0.12 (FAIL)t040: drift 0.112, threshold 0.12 (FAIL)t041: drift 0.0869, threshold 0.12 (PASS)t042: drift 0.0811, threshold 0.12 (PASS)t043: drift 0.0295, threshold 0.12 (PASS)t044: drift 0.024, threshold 0.12 (PASS)t045: drift 0.0159, threshold 0.12 (PASS)t046: drift 0.0655, threshold 0.12 (PASS)t047: drift 0.0463, threshold 0.12 (PASS)t048: drift 0.0451, threshold 0.12 (PASS)t049: drift 0.0737, threshold 0.12 (PASS)t050: drift 0.0854, threshold 0.12 (PASS)suite median 0.0357suite median 0.0357case 1case 50
Each dot is one case's drift; the grey tick beside it is that case's own threshold, measured from its own noise at record time. A dot below its tick is within noise. Colour is the verdict.
● pass ● drift ● fail ─ per-case threshold
●t001json_bare0.03710.12
verdict: FAILdrift 0.0371 vs threshold 0.12noise floor 0.0294judge: not calledcost $0.000972

Baseline

{"category": "billing", "urgency": "medium", "summary": "i was charged twice for my invoice this month, please refund the"}

Current

```json
{"category": "billing", "urgency": "low", "summary": "i was charged twice for my invoice this month, please refund the"}
```
●t0020.02180.12
verdict: PASSdrift 0.0218 vs threshold 0.12noise floor 0.0326judge: not calledcost $0.000488

Baseline

{"category": "billing", "urgency": "low", "summary": "my card was declined but the subscription still shows active. what happened?"}

Current

{"category": "billing", "urgency": "low", "summary": "my card was declined but the subscription still shows active. what happened?"}
●t0030.03390.12
verdict: PASSdrift 0.0339 vs threshold 0.12noise floor 0.022judge: not calledcost $0.000446

Baseline

{"category": "billing", "urgency": "low", "summary": "can you explain the price increase on my last invoice?"}

Current

```json
{"category": "billing", "urgency": "low", "summary": "can you explain the price increase on my last invoice?"}
```
●t0040.040.12
verdict: PASSdrift 0.04 vs threshold 0.12noise floor 0.047judge: not calledcost $0.000473

Baseline

{"category": "billing", "urgency": "low", "summary": "i need a refund for the annual payment, i cancelled two days"}

Current

{"category": "billing", "urgency": "low", "summary": "i need a refund for the annual payment, i cancelled two days"}
●t0050.01710.12
verdict: PASSdrift 0.0171 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000437

Baseline

{"category": "billing", "urgency": "low", "summary": "the invoice pdf shows the wrong billing address."}

Current

{"category": "billing", "urgency": "low", "summary": "the invoice pdf shows the wrong billing address."}
●t0060.02870.12
verdict: PASSdrift 0.0287 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000453

Baseline

{"category": "billing", "urgency": "low", "summary": "why was i billed in usd when my account currency is eur?"}

Current

{"category": "billing", "urgency": "low", "summary": "why was i billed in usd when my account currency is eur?"}
●t0070.0220.12
verdict: PASSdrift 0.022 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000448

Baseline

{"category": "billing", "urgency": "low", "summary": "please send me all invoices for the last financial year."}

Current

{"category": "billing", "urgency": "low", "summary": "please send me all invoices for the last financial year."}
●t0080.01950.12
verdict: PASSdrift 0.0195 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000434

Baseline

{"category": "billing", "urgency": "low", "summary": "i upgraded mid-cycle and the charge looks wrong."}

Current

{"category": "billing", "urgency": "medium", "summary": "i upgraded mid-cycle and the charge looks wrong."}
●t0090.01390.12
verdict: PASSdrift 0.0139 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000477

Baseline

{"category": "billing", "urgency": "high", "summary": "subscription renewed even though i cancelled. need this refunded asap."}

Current

{"category": "billing", "urgency": "high", "summary": "subscription renewed even though i cancelled. need this refunded asap."}
●t0100.01850.12
verdict: PASSdrift 0.0185 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000450

Baseline

{"category": "billing", "urgency": "low", "summary": "do you support payment by bank transfer instead of card?"}

Current

{"category": "billing", "urgency": "low", "summary": "do you support payment by bank transfer instead of card?"}
●t0110.03510.12
verdict: PASSdrift 0.0351 vs threshold 0.12noise floor 0.0323judge: not calledcost $0.000465

Baseline

{"category": "bug", "urgency": "medium", "summary": "the export button crashes the whole page with a 500 error every"}

Current

{"category": "bug", "urgency": "low", "summary": "the export button crashes the whole page with a 500 error every"}
●t0120.03250.12
verdict: PASSdrift 0.0325 vs threshold 0.12noise floor 0.0491judge: not calledcost $0.000445

Baseline

{"category": "bug", "urgency": "medium", "summary": "uploading a csv over 10mb fails silently, nothing happens."}

Current

{"category": "bug", "urgency": "low", "summary": "uploading a csv over 10mb fails silently, nothing happens."}
●t0130.02560.12
verdict: PASSdrift 0.0256 vs threshold 0.12noise floor 0.0223judge: not calledcost $0.000439

Baseline

{"category": "bug", "urgency": "low", "summary": "dark mode is broken on safari, the text is invisible."}

Current

{"category": "bug", "urgency": "medium", "summary": "dark mode is broken on safari, the text is invisible."}
●t0140.01760.12
verdict: PASSdrift 0.0176 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000441

Baseline

{"category": "bug", "urgency": "low", "summary": "app freezes when i open the settings panel on mobile."}

Current

{"category": "bug", "urgency": "low", "summary": "app freezes when i open the settings panel on mobile."}
●t0150.01930.12
verdict: PASSdrift 0.0193 vs threshold 0.12noise floor 0.0222judge: not calledcost $0.000446

Baseline

{"category": "bug", "urgency": "low", "summary": "i get an exception trace whenever i try to share a report."}

Current

{"category": "bug", "urgency": "low", "summary": "i get an exception trace whenever i try to share a report."}
●t0160.08590.12
verdict: PASSdrift 0.0859 vs threshold 0.12noise floor 0.0163judge: not calledcost $0.000467

Baseline

{"category": "other", "urgency": "low", "summary": "search returns no results even for documents i can see in the"}

Current

```json
{"category": "bug", "urgency": "low", "summary": "search returns no results even for documents i can see in the"}
```
●t0170.06180.12
verdict: PASSdrift 0.0618 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000463

Baseline

{"category": "other", "urgency": "low", "summary": "the webhook fires twice for every event, causing duplicate records."}

Current

{"category": "bug", "urgency": "low", "summary": "the webhook fires twice for every event, causing duplicate records."}
●t0180.08520.12
verdict: PASSdrift 0.0852 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000463

Baseline

{"category": "other", "urgency": "low", "summary": "timestamps display in the wrong timezone after the last update."}

Current

```json
{"category": "feature_request", "urgency": "low", "summary": "timestamps display in the wrong timezone after the last update."}
```
●t0190.06970.12
verdict: PASSdrift 0.0697 vs threshold 0.12noise floor 0.0217judge: not calledcost $0.000460

Baseline

{"category": "other", "urgency": "high", "summary": "everything is down, the dashboard will not load at all. urgent."}

Current

{"category": "billing", "urgency": "high", "summary": "everything is down, the dashboard will not load at all. urgent."}
●t020judge: better0.09970.12
verdict: PASSdrift 0.0997 vs threshold 0.12noise floor 0.0247judge: bettercost $0.000428

Baseline

{"category": "other", "urgency": "low", "summary": "pagination skips the last row on every page."}

Current

{"category": "bug", "urgency": "low", "summary": "pagination skips the last row on every page."}
●t0210.00670.12
verdict: PASSdrift 0.0067 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000466

Baseline

{"category": "feature_request", "urgency": "low", "summary": "would be nice if you could add bulk delete for archived items."}

Current

{"category": "feature_request", "urgency": "low", "summary": "would be nice if you could add bulk delete for archived items."}
●t0220.03630.12
verdict: PASSdrift 0.0363 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000418

Baseline

{"category": "feature_request", "urgency": "low", "summary": "please add support for sso with okta."}

Current

{"category": "feature_request", "urgency": "medium", "summary": "please add support for sso with okta."}
●t023systemic drift + judge: worse0.10440.12
verdict: FAILdrift 0.1044 vs threshold 0.12noise floor 0.0251judge: worsecost $0.000429

Baseline

{"category": "other", "urgency": "low", "summary": "any chance of a dark mode for the mobile app?"}

Current

```json
{"category": "bug", "urgency": "low", "summary": "any chance of a dark mode for the mobile app?"}
```
●t0240.00860.12
verdict: PASSdrift 0.0086 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000442

Baseline

{"category": "feature_request", "urgency": "low", "summary": "i wish there was a way to schedule reports weekly."}

Current

```json
{"category": "feature_request", "urgency": "low", "summary": "i wish there was a way to schedule reports weekly."}
```
●t0250.02470.12
verdict: PASSdrift 0.0247 vs threshold 0.12noise floor 0.0225judge: not calledcost $0.000434

Baseline

{"category": "feature_request", "urgency": "low", "summary": "suggestion: let us tag teammates in comments."}

Current

{"category": "feature_request", "urgency": "medium", "summary": "suggestion: let us tag teammates in comments."}
●t0260.00810.12
verdict: PASSdrift 0.0081 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000442

Baseline

{"category": "feature_request", "urgency": "low", "summary": "can you add a slack integration for notifications?"}

Current

{"category": "feature_request", "urgency": "low", "summary": "can you add a slack integration for notifications?"}
●t0270.07790.12
verdict: PASSdrift 0.0779 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000459

Baseline

{"category": "other", "urgency": "low", "summary": "it would help a lot to have keyboard shortcuts for navigation."}

Current

{"category": "billing", "urgency": "low", "summary": "it would help a lot to have keyboard shortcuts for navigation."}
●t0280.02490.12
verdict: PASSdrift 0.0249 vs threshold 0.12noise floor 0.0214judge: not calledcost $0.000443

Baseline

{"category": "feature_request", "urgency": "low", "summary": "please add an api endpoint for listing workspaces."}

Current

{"category": "feature_request", "urgency": "low", "summary": "please add an api endpoint for listing workspaces."}
●t0290.01840.12
verdict: PASSdrift 0.0184 vs threshold 0.12noise floor 0.0157judge: not calledcost $0.000456

Baseline

{"category": "account", "urgency": "low", "summary": "i am locked out after too many login attempts, please reset."}

Current

{"category": "account", "urgency": "low", "summary": "i am locked out after too many login attempts, please reset."}
●t0300.0390.12
verdict: PASSdrift 0.039 vs threshold 0.12noise floor 0.0248judge: not calledcost $0.000432

Baseline

{"category": "account", "urgency": "low", "summary": "how do i change the email address on my account?"}

Current

{"category": "account", "urgency": "medium", "summary": "how do i change the email address on my account?"}
●t0310.03170.12
verdict: PASSdrift 0.0317 vs threshold 0.12noise floor 0.0279judge: not calledcost $0.000421

Baseline

{"category": "account", "urgency": "high", "summary": "i lost my 2fa device and cannot sign in."}

Current

```json
{"category": "account", "urgency": "high", "summary": "i lost my 2fa device and cannot sign in."}
```
●t0320.01860.12
verdict: PASSdrift 0.0186 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000435

Baseline

{"category": "account", "urgency": "low", "summary": "please delete my account and all associated data."}

Current

```json
{"category": "account", "urgency": "medium", "summary": "please delete my account and all associated data."}
```
●t033json_bare0.03630.12
verdict: FAILdrift 0.0363 vs threshold 0.12noise floor 0.0266judge: not calledcost $0.000828

Baseline

{"category": "account", "urgency": "low", "summary": "my password reset email never arrives."}

Current

{"category": "account", "urgency": "low", "summary": "my password reset email never arrives."}
●t0340.09390.12
verdict: PASSdrift 0.0939 vs threshold 0.12noise floor 0.0335judge: not calledcost $0.000458

Baseline

{"category": "other", "urgency": "medium", "summary": "i need to transfer ownership of the workspace to a colleague."}

Current

{"category": "billing", "urgency": "low", "summary": "i need to transfer ownership of the workspace to a colleague."}
●t035judge: tie0.13360.12
verdict: DRIFTdrift 0.1336 vs threshold 0.12noise floor 0.0376judge: tiecost $0.000444

Baseline

{"category": "other", "urgency": "low", "summary": "can i merge two accounts that use different emails?"}

Current

{"category": "billing", "urgency": "low", "summary": "can i merge two accounts that use different emails?"}
●t0360.03360.12
verdict: PASSdrift 0.0336 vs threshold 0.12noise floor 0.047judge: not calledcost $0.000434

Baseline

{"category": "account", "urgency": "low", "summary": "login redirects me in a loop and never completes."}

Current

{"category": "account", "urgency": "low", "summary": "login redirects me in a loop and never completes."}
●t037systemic drift + judge: worse0.110.12
verdict: FAILdrift 0.11 vs threshold 0.12noise floor 0.0232judge: worsecost $0.000455

Baseline

{"category": "other", "urgency": "low", "summary": "just wanted to say the new release is great, thanks team."}

Current

{"category": "account", "urgency": "low", "summary": "just wanted to say the new release is great, thanks team."}
●t0380.0860.12
verdict: PASSdrift 0.086 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000427

Baseline

{"category": "other", "urgency": "low", "summary": "do you have a status page i can subscribe to?"}

Current

{"category": "billing", "urgency": "low", "summary": "do you have a status page i can subscribe to?"}
●t039systemic drift + judge: worse0.11080.12
verdict: FAILdrift 0.1108 vs threshold 0.12noise floor 0.024judge: worsecost $0.000440

Baseline

{"category": "other", "urgency": "low", "summary": "where can i find your data processing agreement?"}

Current

{"category": "feature_request", "urgency": "low", "summary": "where can i find your data processing agreement?"}
●t040json_bare0.1120.12
verdict: FAILdrift 0.112 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000855

Baseline

{"category": "other", "urgency": "low", "summary": "are you hiring for backend roles right now?"}

Current

```json
{"category": "billing", "urgency": "low", "summary": "are you hiring for backend roles right now?"}
```
●t0410.08690.12
verdict: PASSdrift 0.0869 vs threshold 0.12noise floor 0.0392judge: not calledcost $0.000431

Baseline

{"category": "other", "urgency": "medium", "summary": "what is your uptime guarantee on the business plan?"}

Current

{"category": "billing", "urgency": "low", "summary": "what is your uptime guarantee on the business plan?"}
●t0420.08110.12
verdict: PASSdrift 0.0811 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000410

Baseline

{"category": "other", "urgency": "low", "summary": "is there a student discount available?"}

Current

{"category": "account", "urgency": "low", "summary": "is there a student discount available?"}
●t0430.02950.12
verdict: PASSdrift 0.0295 vs threshold 0.12noise floor 0.0327judge: not calledcost $0.000454

Baseline

{"category": "billing", "urgency": "low", "summary": "the payment page throws an error when i try to update my"}

Current

{"category": "billing", "urgency": "low", "summary": "the payment page throws an error when i try to update my"}
●t0440.0240.12
verdict: PASSdrift 0.024 vs threshold 0.12noise floor 0.0171judge: not calledcost $0.000471

Baseline

{"category": "billing", "urgency": "low", "summary": "i was charged but the app crashed before confirming, did it go"}

Current

```json
{"category": "billing", "urgency": "low", "summary": "i was charged but the app crashed before confirming, did it go"}
```
●t0450.01590.12
verdict: PASSdrift 0.0159 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000435

Baseline

{"category": "account", "urgency": "low", "summary": "login is broken after i changed my email address."}

Current

{"category": "account", "urgency": "low", "summary": "login is broken after i changed my email address."}
●t0460.06550.12
verdict: PASSdrift 0.0655 vs threshold 0.12noise floor 0.0judge: not calledcost $0.000467

Baseline

{"category": "other", "urgency": "low", "summary": "the export is so slow it is basically unusable, can you make"}

Current

{"category": "billing", "urgency": "low", "summary": "the export is so slow it is basically unusable, can you make"}
●t0470.04630.12
verdict: PASSdrift 0.0463 vs threshold 0.12noise floor 0.0625judge: not calledcost $0.000409

Baseline

{"category": "billing", "urgency": "high", "summary": "my invoice download link returns a 500."}

Current

{"category": "bug", "urgency": "high", "summary": "my invoice download link returns a 500."}
●t0480.04510.12
verdict: PASSdrift 0.0451 vs threshold 0.12noise floor 0.0466judge: not calledcost $0.000446

Baseline

{"category": "billing", "urgency": "high", "summary": "cannot sign in and i am being billed for a seat i"}

Current

{"category": "billing", "urgency": "high", "summary": "cannot sign in and i am being billed for a seat i"}
●t0490.07370.12
verdict: PASSdrift 0.0737 vs threshold 0.12noise floor 0.0618judge: not calledcost $0.000462

Baseline

{"category": "billing", "urgency": "low", "summary": "there is no way to cancel a subscription in the ui, please"}

Current

{"category": "billing", "urgency": "low", "summary": "there is no way to cancel a subscription in the ui, please"}
●t0500.08540.12
verdict: PASSdrift 0.0854 vs threshold 0.12noise floor 0.0242judge: not calledcost $0.000424

Baseline

{"category": "other", "urgency": "low", "summary": "following up on my previous ticket, any update?"}

Current

{"category": "bug", "urgency": "low", "summary": "following up on my previous ticket, any update?"}