Skip to content

Evaluation and Optimization

This example improves one extraction prompt with supervised examples and a metric that returns both a score and actionable feedback.

Complete Program

> adapter: pydantic
> model: groq:qwen/qwen3-32b

@extract_owner(ticket: str) -> str:
  > optimize: owner_prompt:
    Read the support ticket below.
    Ticket: <ticket>
    Return only the team that owns the next action: [owner]
  = `owner`

@eval: extract_owner:
  > data: tickets:
    = ```
    return [
      ("Payments: retries fail after authorization", {"owner": "Payments"}),
      ("Search indexing is twelve hours behind", {"owner": "Search"}),
      ("Identity reports invalid refresh tokens", {"owner": "Identity"}),
    ]
    ```

  > test_data: tickets:
    = ```
    return [
      ("Checkout cannot create payment intents", {"owner": "Payments"}),
      ("New documents do not appear in results", {"owner": "Search"}),
    ]
    ```

  > metric: exact_owner(tickets):
    = ```
    actual = extract_owner(tickets).strip().casefold()
    wanted = expected["owner"].casefold()
    if actual == wanted:
        return 1.0, None
    return 0.0, f"returned {actual!r}; expected {wanted!r}"
    ```

= `extract_owner("Identity: sessions expire immediately after login")`

The bare prompt lines inside > optimize: are intentional. They are legacy template syntax accepted only inside > optimize: and > auto: bodies. They are newline-joined into one model call, exactly like:

@extract_owner_explicit(ticket: str) -> str:
  > optimize: owner_prompt:
    >> Read the support ticket below.
    Ticket: <ticket>
    Return only the team that owns the next action: [owner]
  = `owner`

Do not add >> to each continuation line. Bare template lines outside these two special directives are parse errors.

Run Tests and Evals

kedi program.kedi --eval

Normal evaluation uses matching test_data when present; otherwise it falls back to data. It reports one selected score, not separate training and test scores.

Run GEPA

kedi program.kedi \
  --optimize \
  --optimizer gepa \
  --optimizer-max-metric-calls 30 \
  --eval

GEPA trains on data, validates against matching test_data, and stores only an optimized prefix. The source prompt and its output schema remain authoritative.

Generated files are:

  • program.kedi.optimized.json for selected prompt prefixes;
  • program.kedi.optimized_scores.json for score metadata;
  • program.kedi.gepa/ for per-span checkpoints.

Use --optimizer-fresh to remove these optimization artifacts before a new GEPA run. --no-cache instead controls generated > auto: code and does not reset optimizer state.

Dataset Shape

Explicit (input, expected_dict) rows are the least ambiguous format across evaluation and optimization. For a procedure with multiple parameters, use ((arg1, arg2), expected_dict). Metrics should be deterministic, stable, and cheap compared with the model call; vague feedback gives the optimizer little usable evidence.