How Strop works¶
The three partitions¶
strop data split shuffles your rows with a seed and stores the result, so the same data and
settings always give the same split.
| Partition | Used for |
|---|---|
train |
Finding failures. The analyzer and the optimizer only see failed train examples. |
validation |
Comparing prompts. Every candidate is scored on the same validation rows. |
test |
One final check with strop test --confirm. |
The test partition is locked in code, not only in the command line:
- The evaluation service refuses the test partition unless it was created for
strop test. - The optimizer and analysis packages have no access to partitions or storage. They receive a fixed view of train failures and validation scores. A test checks that they cannot import either.
strop testasks for--confirmand refuses a second run without--allow-rerun, because tuning the prompt after looking at the test result makes the result optimistic.
The split is saved with the dataset's hash. If the data file changes later, Strop asks you to split again instead of mixing old and new rows.
One turn of the loop¶
strop optimize --max-turns N repeats the following up to N times:
- Take the best prompt so far and score it on validation. A stored result is reused when the prompt, model, output contract, and metrics have not changed.
- Run it on the train rows and classify every failure.
- Ask the analyzer for failure patterns. These are hypotheses, and each must cite example ids.
- Ask the optimizer for
--candidatesrevised prompts. Each reply must be valid JSON and must cite example ids. A reply that edits a frozen message, drops a{{variable}}, changes a<strop:fixed>span, or does not change the prompt is discarded and recorded with its reason. - Score each valid candidate on the validation rows.
- Apply the acceptance rules. Of the candidates that pass, the one with the largest gain wins.
- The winner becomes the best prompt (or waits for approval), and the next turn starts from it.
Every candidate is stored as a prompt version, including rejected and invalid ones, together
with its summary, evidence, and the exact reason for the decision. strop history lists them.
Acceptance rules¶
A candidate becomes the best prompt only if all of these hold:
- The validation partition has at least
min_validation_samplesrows. - The primary metric improves by at least
min_improvement. - Every constraint holds. If the best prompt already violates a constraint, the candidate may not make it worse.
- The valid output rate does not fall by more than
max_validity_regression. - No class loses more than
max_class_regressionof its F1. - With
require_paired_wins, the candidate wins on more validation examples than it loses.
If a candidate fails, every failed rule is listed in its reason. The best prompt stays as it was.
Approval and resuming¶
With --approval manual, the winning candidate is stored as pending and the command exits with
code 10 after printing the next steps. Review it with strop diff v4, then run
strop approve v4 or strop reject v4, and continue with strop optimize --resume.
A run that is stopped by a budget (exit code 11) or by Ctrl-C (exit code 12) is saved after every
candidate. strop optimize --resume continues the unfinished turn without repeating work that
is stored or cached.
When a run stops¶
The final report always says why:
| Reason | Meaning |
|---|---|
max_turns |
The turn limit was reached. |
target_reached |
The primary metric reached --target on validation. |
patience |
--patience turns in a row produced no accepted prompt. |
no_failures |
The best prompt has no failures left on the train rows. |
budget |
The call, cost, or time budget was reached. |
awaiting_approval |
A candidate is waiting for strop approve or strop reject. |
When a run ends, Strop prints the metric before and after and a colored diff between the initial prompt (v1) and the final best prompt.
Prompt versions¶
The prompt file is registered as v1 on the first run. Versions are immutable: a database
trigger refuses updates and deletes. strop rollback v2 only moves the pointer to the best
version. Your own edits are added with strop prompt add --file edited.yaml --use. If you edit
the prompt file without registering it, Strop warns that it differs from every stored version.
Caching and repeatability¶
- Responses at temperature 0 are cached by model, messages, and settings. Responses at a higher temperature are never cached, so repeated sampling stays varied.
--no-cacheskips the cache for one command.strop clean --cache --yesempties it.strop baseline --repeats Nruns the baseline N times; runs after the first skip the cache and report how much the primary metric moved between runs.- Each stored evaluation records the model and a fingerprint of the settings that affect it.
Local data¶
Everything lives in your project directory.
| Path | Content |
|---|---|
strop.yaml |
Your configuration. |
.strop/strop.db |
SQLite database: splits, prompt versions, evaluations, per-example results, runs, candidates, cached responses. |
exports/ |
Written by strop export. |
.strop/ contains model outputs and cached responses for your dataset rows. Keep it, and your
.env file, out of version control. API keys are read from the environment or from .env and
are never written to the database, logs, or reports.
Failure handling¶
- A failed model call is recorded as
provider_errorfor that example and does not stop the run. Temporary errors (rate limits, timeouts, outages) are retried with exponential backoff. - Rate-limit pressure is bounded by
max_concurrency. - Budgets are checked between batches, so a run can finish the current batch before it stops.