Failure Analysis

Failure analysis turns a wall of stack traces into a labelled cause per test. Every failed test is matched against seven root-cause categories and tagged with a root-cause chip right beside its failure detail in the suite breakdown, so you can see what broke rather than scrolling through every individual trace.

• Seven categories: each failure is sorted into one of seven trained root-cause buckets: assertion failures, timeouts, element-not-found, network errors, navigation errors, environment/config issues, and flaky-passed-on-retry. Anything the classifier can't place with confidence falls back to unknown.
• Per-test chip: each failed test gets its own chip showing the category and a representative first line of the error, inline with the failure detail.
• Match strength: each chip carries a qualitative strong / moderate / weak label, never a numeric confidence percentage.

Classification runs in-process: a compact classifier embedded in the package, trained offline and shipped inside it, optionally augmented by a second layer trained on your own machine from failures you label (see On-device training). It never sends your stack traces, test names, or any other data over the network, and it does not call any AI or LLM service; training, evaluation, and the resulting model file all stay local. The report shows a qualitative match strength (strong, moderate, or weak) rather than a raw probability, because a precise-looking percentage would overstate what a lightweight model should promise.

How much detail you see depends on the template:

• detailed: a root-cause chip under each failed test in the suite breakdown: its category, match strength, and a representative error line.
• executive: a single one-liner summarising the dominant root cause (no per-test chips).
• minimal: nothing; the section is omitted entirely.

Failure analysis is on by default. Pair it with the detailed template to see the full breakdown.

reporter: [['@reportforge/playwright-pdf', {  template: 'detailed',  failureAnalysis: { enabled: true, maxClusters: 5 },}]]

To turn the section off entirely, set failureAnalysis: { enabled: false }. Your tests and PDF are otherwise unaffected.

When a failure can't be recognised, or is matched only with weak confidence, the reporter records it locally; these samples are the raw material for on-device training. They are appended to ~/.reportforge/{project}/unclassified.csv with an empty label column for you to fill.

• Tokenized and redacted: the stored text is reduced to tokens with URLs, email addresses, high-entropy secrets, IPs, and filesystem paths stripped out. Raw error text is never written.
• Local only: the file stays on your machine. Nothing is ever transmitted, uploaded, or phoned home.
• First-run notice: a one-time notice is printed the first time anything is collected, so the behaviour is never silent.
• Labels survive: a labeled row is never overwritten when the same failure recurs, and labeled rows are exempt from the bounded local buffer's eviction.

Label the collected samples interactively:

npx @reportforge/playwright-pdf label-feedback

Each answer is saved immediately, so quitting mid-session loses nothing. Prefer a text editor? Fill the label column by hand with one of the seven category names. By default only unrecognized/low-confidence failures are collected; set collectScope: 'all' to also collect confidently-classified failures so wrong-but-confident predictions can be corrected too. Opt out of collection entirely with collectUnclassified: false.

To review or move your training data between machines, merge every project's file into a single CSV in the current directory (a local file; there is no upload):

npx @reportforge/playwright-pdf export-feedback

The base classifier ships trained on a curated public corpus; it has never seen your app's failure vocabulary. Once you have labeled at least 10 rows across 2+ categories, train a personal layer entirely on your machine:

npx @reportforge/playwright-pdf train-model

Training is a small statistical fit (milliseconds, no GPU, no Python, no network). What makes it safe to trust is the evaluation gate that runs in the same command:

• Cross-validated: every labeled row is scored by a model that never saw it, so the gate measures real generalization rather than memorization.
• Calibrated: the local layer only answers when its measured precision on your own held-out rows is high; below that confidence, the base model answers.
• Gated activation: the layer activates only when it measurably beats the base model on your own labeled set. Mislabeled or noisy data degrades to a no-op, never to worse classification.
• Rules still win: the deterministic high-precision rules (flaky-on-retry, connection errors, strict-mode violations, …) always outrank both models.

The result is written to ~/.reportforge/model-local.json and picked up automatically on the next run. Compare base vs. base+local any time without writing anything:

npx @reportforge/playwright-pdf evaluate-model

Share with CI / your team: the model file contains only redacted token statistics (never raw failure text), so it is safe to commit. Train with train-model --out=./reportforge-model.json --project (the --project flag keeps other local projects' feedback out of the committed file) and point the reporter at it:

reporter: [['@reportforge/playwright-pdf', {  failureAnalysis: { localModelPath: './reportforge-model.json' },}]]

Opt out of the local layer entirely with localModel: false; the reporter then behaves exactly as before this feature existed.

Two model layers, two very different update paths, and only one of them ever involves a server, in the inbound direction only:

The base model (shipped, signed)

• Auto-updates: when a newer base model is available, it reaches the reporter automatically with no separate download and no extra network calls, and is cached locally for subsequent runs.
• Signed and verified: every delivered model is cryptographically signed and verified on-device. The signature is re-checked on every load, so a tampered cache file is rejected and never used.
• Fully offline-tolerant: there is no network dependency. The bundled model is always the floor: if there is no cached model, no network, or the cached model fails verification, the reporter falls back to the model shipped in the package.
• Monotonic: a delivered model is only adopted when its version is newer than the bundled model; older or equal versions are ignored.

Your local layer (trained on-device)

• Trained, evaluated, stored, and consumed on your machine. It is never uploaded, never delivered by the server, and never signed: it is your artifact.
• Independent of base updates: a new base model does not deactivate your layer. The training CLI records which base version the gate was checked against and warns when it has since changed, so you can re-run train-model to re-validate.

To pin the bundled base model for reproducibility or audit, set autoUpdateModel: false. The reporter then always uses the model shipped in the package and ignores any cached update:

reporter: [['@reportforge/playwright-pdf', {  template: 'detailed',  failureAnalysis: { autoUpdateModel: false },}]]

The complete inventory of what the reporter sends anywhere, so the failure-analysis guarantee is auditable rather than a slogan:

All keys live under the failureAnalysis object.