Configuration reference¶
strop init writes strop.yaml in your project directory. Edit it by hand any time. Strop
rejects unknown keys, so a typo is reported instead of ignored. Command-line flags override the
values in this file for a single run.
Paths in strop.yaml are relative to the project directory.
Complete example¶
project:
name: ticket-routing
task_description: Route a customer message to a support category
prompt:
file: prompt.yaml
dataset:
path: data.csv
id_column: ticket_id
columns:
customer_message: input
category: expected
channel: metadata
output:
type: label
labels: [billing, refund, exchange, technical, other]
models:
target:
model: <provider/model>
temperature: 0.0
optimizer:
model: <provider/model>
metrics:
primary: macro_f1
secondary: [accuracy]
constraints:
valid_output_rate: {min: 0.98}
recall:refund: {min: 0.80}
split:
train: 0.6
validation: 0.2
test: 0.2
seed: 42
optimization:
max_turns: 5
candidates_per_turn: 3
patience: 2
approval: auto
budgets:
max_total_model_calls: 500
max_wall_time_minutes: 60
execution:
batch_size: 10
max_concurrency: 3
retry_attempts: 3
Only project, dataset, output, models.target, and metrics.primary are required. Every
other value has the default shown in the tables below.
project¶
| Key | Default | Meaning |
|---|---|---|
name |
required | Project name. |
task_description |
required | One sentence describing what the prompt must do. The optimizer and the judge see it. |
prompt¶
| Key | Default | Meaning |
|---|---|---|
file |
prompt.yaml |
The starting prompt. See Prompt files. |
dataset¶
| Key | Default | Meaning |
|---|---|---|
path |
required | CSV or JSONL file. |
format |
from the extension | csv or jsonl. Extensions .csv, .jsonl, and .ndjson are recognized. |
id_column |
none | Column with a unique id per row. Without it, rows are numbered by position. |
columns |
required | Maps column names to roles: input, expected, metadata, or ignore. |
Rules for columns:
- At least one column must be
input. Input column names must be usable as prompt variables: letters, digits, and underscores, not starting with a digit. - At most one column may be
expected. It is required by metrics that compare with a reference (accuracy, F1, exact match, field metrics, similarity). metadatacolumns are kept with the row but never sent to a model.- Columns that are not listed are ignored.
Files must be UTF-8. JSONL values that are not strings are stored as JSON text, so an expected
JSON object in a JSONL file works for json outputs.
Invalid rows (empty inputs, empty expected values, labels outside output.labels, expected
values that are not valid JSON for json outputs, duplicate ids) are reported with their row
numbers. Commands stop on invalid rows unless you pass --skip-invalid to strop data split.
output¶
| Key | Default | Meaning |
|---|---|---|
type |
required | label, json, or text. |
labels |
none | Allowed labels. Required for label (at least two, no duplicates). |
schema_file |
none | JSON Schema file the output must satisfy. Only for json. |
match |
strict |
For label only. strict accepts the label alone, ignoring case, quotes, and a final period. contains also accepts an output that contains exactly one label as a whole word. |
An output that does not fit the contract is recorded as an invalid output. It is never dropped.
For json, a JSON object inside a Markdown code fence is accepted.
models¶
Each role takes a model name that LiteLLM understands, such as <provider>/<model>. Strop has no
default model.
| Role | Required | Purpose |
|---|---|---|
target |
yes | The model the prompt is optimized for. |
optimizer |
for optimize |
Proposes revised prompts. |
analyzer |
no | Proposes failure patterns. Falls back to optimizer. |
judge |
for judge metrics | Scores judge rubrics. |
Settings for each role:
| Key | Default | Meaning |
|---|---|---|
model |
required | Model name. |
temperature |
0.0 |
Between 0 and 2. Responses are only cached at temperature 0. |
max_tokens |
none | Output token limit. |
timeout_s |
60 |
Seconds to wait for one call. |
params |
{} |
Extra keyword arguments passed to LiteLLM, for example api_base. |
Parameters a model does not support are dropped instead of causing an error.
metrics¶
| Key | Default | Meaning |
|---|---|---|
primary |
required | The metric the loop tries to improve. |
secondary |
[] |
Metrics that are reported but do not decide anything. |
constraints |
{} |
Limits a candidate must respect, as name: {min: x} or {max: x}. |
rules |
{} |
Deterministic output checks. Each becomes the metric rule:<name>. |
python |
none | A custom metric file. It becomes the metric custom. |
judges |
{} |
Judge rubrics. Each becomes the metric judge:<name>. |
The metrics valid_output_rate, every rule, custom, and every judge are always reported.
See Metrics for all metric names.
rules¶
metrics:
rules:
short: {type: max_words, value: 120}
polite: {type: not_matches, value: "(?i)stupid|idiot"}
complete: {type: contains_all, value: [refund, days]}
| Type | Value | Passes when |
|---|---|---|
max_chars, min_chars |
whole number | the output has at most or at least that many characters |
max_words, min_words |
whole number | the output has at most or at least that many words |
matches |
regular expression | the expression is found in the output |
not_matches |
regular expression | the expression is not found in the output |
contains_all |
list of strings | every string is present, ignoring case |
contains_none |
list of strings | no string is present, ignoring case |
Rules check the raw model output. An empty output fails every rule. Outputs lost to provider errors are not scored.
python¶
metrics:
python:
file: metric.py
timeout_s: 120
direction: maximize
metric.py defines score(output, row). output is the raw model text. row is a dictionary
with inputs (the input columns), expected (or None), and value (the parsed output, or
None if it was invalid). Return a number from 0 to 1, a bool, or
{"score": 0.5, "feedback": "short explanation"}. The feedback is shown to the optimizer.
The file runs in a separate Python process with a timeout. It is your own code, so Strop runs it as written. Dataset content is passed to it as data and is never executed. Anything the file prints is ignored.
judges¶
metrics:
judges:
helpful:
criteria: The answer gives the customer a clear next step.
scale_min: 1
scale_max: 5
instructions: ""
A judge model scores each output as an integer from scale_min to scale_max and gives a short
reason. Scores are normalized to 0 to 1. Judge scores are estimates and are labeled as subjective
in every report. Judge metrics need models.judge.
split¶
| Key | Default | Meaning |
|---|---|---|
train |
0.6 |
Share of rows used for failure analysis and optimizer evidence. |
validation |
0.2 |
Share of rows used to compare prompts. |
test |
0.2 |
Share of rows kept for the final check. May be 0. |
seed |
42 |
Seed for the shuffle. |
stratify |
true |
Keep class proportions the same in every partition. |
The ratios must sum to 1. Stratifying only applies when the expected values are class-like. If most expected values are unique (free text, JSON objects, numbers), the split is random.
optimization¶
| Key | Default | Meaning |
|---|---|---|
max_turns |
5 |
Turns per run. |
candidates_per_turn |
3 |
Revised prompts proposed per turn. |
target |
none | Stop when the primary metric reaches this value. |
patience |
2 |
Stop after this many turns in a row without an accepted prompt. |
approval |
auto |
auto promotes the winner. manual pauses for strop approve or strop reject. |
min_improvement |
0.01 |
Smallest primary metric gain a candidate needs. |
train_sample |
none | Analyze a seeded sample of this many train rows per turn. |
instructions |
"" |
Extra guidance for the optimizer. |
evidence_examples |
12 |
Failed examples shown to the analyzer and optimizer. |
evidence_chars |
400 |
Each text field of an example is cut to this length. |
max_class_regression |
0.10 |
Largest allowed drop in any class F1. null turns the check off. |
max_validity_regression |
0.02 |
Largest allowed drop in the valid output rate. |
min_validation_samples |
10 |
Fewest validation rows needed to compare prompts. |
require_paired_wins |
true |
The candidate must win on more examples than it loses. |
budgets¶
| Key | Default | Meaning |
|---|---|---|
max_total_model_calls |
500 |
Fresh model calls per run. Cached answers are free and are not counted. null turns the limit off. |
max_total_cost_usd |
none | Cost limit. Only calls that report a cost count towards it. |
max_wall_time_minutes |
60 |
Time limit per run. null turns the limit off. |
execution¶
| Key | Default | Meaning |
|---|---|---|
batch_size |
10 |
Rows per batch. Budgets are checked between batches. |
max_concurrency |
3 |
Simultaneous model calls. |
retry_attempts |
3 |
Attempts for a call that fails with a temporary error. |
Prompt files¶
A prompt file is YAML with chat messages:
messages:
- role: system
content: |-
You route customer messages to a support team.
Reply with the category only.
- role: user
content: |-
Message: {{customer_message}}
optimize:
- system
- Roles are
system,user, andassistant. {{name}}is replaced by the input column calledname. Every variable must match an input column.optimizelists the roles the optimizer may rewrite. Other messages are never changed. It defaults tosystemif there is a system message, otherwiseuser.- Wrap text that must never change in
<strop:fixed>...</strop:fixed>. The optimizer must keep it exactly, and the markers are removed before the prompt is sent to a model. - A file ending in
.mdor.txtis treated as a single user message that the optimizer may rewrite.
Your prompt file is never overwritten. Every revision is stored as a new immutable version in
.strop/.