Abstract
Most assistant evaluations measure performance on a fixed task. Real work rarely stays fixed: priorities change, requirements collide, and an earlier decision must be revisited. Constraint Shift Benchmarks evaluates whether an assistant can adapt without losing the user's original goal.
The problem
A capable assistant should not simply follow the newest instruction in isolation. It should recognize when a new constraint changes the plan, explain the trade-off, and retain the details that remain relevant.
We test three failure modes:
- Context loss — earlier requirements silently disappear.
- Constraint masking — an impossible combination is treated as achievable.
- Over-correction — a new request causes unrelated work to be rewritten.
Benchmark design
Each task begins with a short, realistic workflow: drafting a release, diagnosing a deployment, or planning an analysis. We then introduce a single change in scope, timing, or policy.
| Capability | What we measure |
|---|---|
| Goal retention | Does the assistant preserve valid earlier requirements? |
| Conflict detection | Does it identify genuinely incompatible constraints? |
| Change isolation | Does it modify only the work affected by the new information? |
| Communication | Does it explain the consequence in clear, actionable language? |
Example intervention
Initial request: Publish the report after legal review.
New constraint: The report must be live today.
A strong response does not invent approval or quietly publish. It identifies the conflict, states the decision needed, and keeps the report ready for the authorized next step.
Scoring
We score each response with a paired rubric: one reviewer checks task completion, while another checks whether the response handled the changed constraint faithfully. A response must pass both checks to count as successful.
Takeaway
Assistants are most useful when they remain reliable under change. Measuring that reliability requires tests where the task moves — and where preserving the user's intent matters as much as producing fluent output.
Evaluation · 2026 · 08