A successful ending can follow a difficult conversation. A long conversation can be careful persistence—or a system that failed to stop. The useful comparison is the one that leaves those differences inspectable.
This readout draws on 76 reviewed conversations across 10 assistants. Each model appears in four to nine cases. We report what happened in those assigned records, with the denominator and case mix in view.
Completion is not the only way a run ends.
The current records have two terminal outcomes: successful closure and exhaustion of interest. The rows below use each assistant’s assigned cases as the denominator.
The bars are descriptive rates, not adjusted estimates. Select a row to follow that assistant across the other figures.
Each assistant appears in four to nine assigned cases. Rates can move sharply when one case changes; compare the count beside every percentage before reading a difference as a stable pattern.
One score, with its spread attached.
Mean composite summarizes the case rubric. The range keeps variation visible: two assistants with similar means may have very different observed cases.
The line spans the minimum to maximum observed case score; the dot marks the mean. This is a range, not a confidence interval.
Horizontal position is successful closure rate; vertical position is median assistant turns. Shorter runs are not automatically better, so read duration with outcome and case type.
Sort by a column. Select a model to highlight it in the figures and capability matrix.
How to read: State movement is the mean of per-case proportions of evaluated turns marked present_moved. Red-line cases can overlap either terminal label. Interest Δ is the median per-case change from first to last recorded user-interest value. Neither is a standalone quality score.
Six dimensions of interaction capability.
Scores are averaged across turns within each case, then across cases. This prevents long transcripts from contributing more observations than short ones.
| ASSISTANT · N | L1 Goal | L2 Information | L3 State | L4 Expression | L5 Agency | L6 Horizon |
|---|
Cell color encodes the case-weighted mean on a 0–1 rubric. Exact values remain printed in each cell. Select a model row to link the profile with its outcomes above.
Did user interest rise or fall along the way?
Per-case change in the recorded user-interest signal, from the first to last checkpoint. Positive and negative values describe trajectory; they do not prove why it changed.
Each case contributes one start-to-end difference. The zero line indicates no net change at the recorded checkpoints; the values are model-evaluation signals, not user-study evidence.
These models did not see identical case mixes.
Scenario setting, information sufficiency, and task purpose shape what a case asks of an assistant. Their counts belong beside the results.
Designed case cells across realistic, stylized, and science-fiction settings.
Observed model and scenario combinations in this snapshot.
Different assistants appear in the user role; assignments are not balanced.
Read this as a casebook, not a league table.
One observation is one reviewed conversation. Each record has one assistant under test, a partner in the user role, a scenario, information and purpose labels, assistant-turn count, composite score, terminal state, and per-turn process signals. The published aggregates are descriptive summaries of the assigned cases.
Current outcome.composite on a 0–1 scale, then averaged by assistant. Current capped level means are used throughout; archived original_scores are not used for rankings. It is not a probability and is not interchangeable with another evaluation’s index.
Count of a terminal label divided by that assistant’s assigned case count. All rates retain their raw numerator and denominator.
Median assistant turns. Successful-run turns are conditional on cases that ended in successful closure.
The current outcome.level_means retain the scoring cascade. These case means are averaged for the selected cohort.
State movement, red-line flags, interest, and tolerance come from the current evaluation records. They are not human-subject measurements.
Case counts and partner/scenario mixes differ. There are no repeated runs or uncertainty intervals in this snapshot.
The casebook records rubric scores, terminal outcomes, conversation length, and process signals. Price, output speed, and latency were not recorded. All comparisons use the assigned cases and their observed values.