Reading the report

The report fills the bottom of the run page, under Report, and grows while the run is live. It sorts every finding by what the replay and the judge made of it, so the section at the top is the one to act on.

How a defect is checked

A person reports a defect with the steps that led to it and what they saw. A defect needs at least two steps. People are told to write them as actions only — exact addresses, labels and values — and to keep what went wrong out of them, because the steps are all the replay is told about the defect. A person cannot report a finding before their first action in the browser, or right after the browser itself broke down.

When every person has finished, each defect is replayed:

The replay. A fresh agent opens a new browser at the product’s address and is given the steps — not the title, not what the person saw, not the goal they were pursuing. If the person who reported it had a test account, the replay signs in with the same one. It has 30 steps to follow the instructions, then reports whether it carried out every step or which step it could not do, and what the page showed.
The judge. A model compares the claim with that report and answers:
- confirmed — the replay independently showed the behaviour the claim describes. A step that could not be carried out can itself be the defect, such as a page that failed before its button appeared.
- refuted — the replay showed the opposite, or the thing working.
- inconclusive — the replay does not settle it either way.
A replay that merely repeats an expected result spelled out in the steps, without saying what the page showed, does not count as independent. The judge gets up to two replies to give its answer; a judge that gives none is not taken for inconclusive — the defect goes under Could not be judged, where it can be judged again.

A replay that could not follow the steps far enough to report anything is not judged at all: a failure to replay never refutes a defect.

Friction is never replayed. It is a claim about the person’s experience, not about the product, and there is nothing for a stranger to reproduce.

The sections

Section	Shown	Hint on the page
Confirmed	Always	A fresh agent reproduced it and the judge agreed — empty, it says Nothing confirmed yet. while the run is live and No defect was confirmed. after
Could not be judged	When it has entries	The judge gave no verdict; the replay is kept
Inconclusive	When it has entries	The replay could not settle it
Not judged	When it has entries	Reported, but not replayed and judged to the end — each entry says why
Refuted	When it has entries	The replay did not see the problem
Friction	When it has entries	Not broken, but slowed someone down

Each heading carries a count, such as Confirmed · 2.

A defect is not judged for one of these reasons:

The fresh agent could not follow the steps far enough to report. — the replay ran out of steps, or stopped, without saying what it saw.
Waiting for its replay. or Waiting for the judge. — the run is still live and this defect’s replay or judge has not finished yet.
The run ended before it was replayed. or The run ended before it was judged. — the run ended first: you stopped it, it reached the cap, or its replay failed without handing back a result.

Judging again

A defect under Could not be judged has its replay kept, so only the judge needs to run again. Open it to see Why it was not judged: — Model error: … with the model’s own failure, Failed: … with any other failure, The run’s cap ran out before the judge answered., You stopped the run before the judge answered., The model call was refused before the judge answered., or the error it stopped with.

Once the run has ended, Judge again runs only the judge on the stored replay, without a browser. It is paid from what is left of the run’s cap, uses the run’s model, and needs your email address on the private beta list, like starting a run. While it works the button says Judging again…, the Judge stage shows In progress, and the page keeps refreshing until the answer arrives. The defect then moves to the section its verdict belongs in — or stays, with the new reason, if the judge again gives none.

Instead of the button, a defect may say:

You can judge it again after the run, if its cap has room left. — the run is still live.
This run has spent its cap, so it cannot be judged again.

And Judge again itself can answer:

Message	Why
It is already being judged again.	Someone chose Judge again a moment ago.
Only a defect whose judge gave no verdict can be judged again.	Someone else’s Judge again already settled it.
This run has spent its cap, so it cannot be judged again.	The cap ran out since the page last refreshed.
The workspace model key was removed or changed since this run, so this run’s model cannot be called. Start a new run instead.	The key is now from a different provider or service than the one the run started with.
The judge could not be started. Try again.	Nothing was started; try again.

A finding

Closed, a finding shows its severity — low, medium or high — the person who reported it, and its title; under Not judged and Could not be judged, the reason too. Open it for:

While trying to: the goal the person was working on.
Steps, numbered.
What happened: what the person saw.
What the replay saw: the fresh agent’s own account, for a defect that was replayed.

Put the last two side by side to see why the judge decided as it did.

After the run

When the run has ended, Start another run goes back to the plan. The people start from scratch every time — Trawler carries nothing from one run to the next, and does not compare one run’s findings with another’s yet. Your product, on the other hand, keeps whatever the people changed in it, such as data created under a test account.