History & Flakiness Trending

Two things are built from history data and shown in the detailed template:

• Pass-rate trend: a mini sparkline and delta showing how your overall pass rate changed across the last N runs.
• Flakiness table: top-N tests ranked by flake rate, each with a per-test dot sparkline (amber dot = flaky in that run).

Both sections are controlled by showTrend (default true). They appear once at least two history entries exist.

Zero configuration needed for local development. History is written automatically to ~/.reportforge/{projectKey}/history.json, where projectKey is a short per-project key, so different projects never share a trend.

// playwright.config.tsreporter: [['@reportforge/playwright-pdf', {  template: 'detailed',  // historyFile defaults to ~/.reportforge/{projectKey}/history.json}]]

The sparkline and flakiness table appear automatically once two or more runs have been recorded. No extra configuration required.

Ephemeral CI runners discard the workspace after every run, so the local history file never accumulates and the trend stays stuck at a single data point. The recommended fix is the opt-in server-side trend store:

reporter: [['@reportforge/playwright-pdf', {  template: 'detailed',  remoteHistory: true,}]]

• One small authenticated request per run, nothing extra to wire up. The server keeps roughly the last 100 runs per project and branch (expiring after 180 days of inactivity) and returns the most recent 50 for the chart. Zero pipeline config.
• Aggregate numbers only: pass rate, counts, duration, verdict, an opaque run id. No test titles, no error text; even the branch name never travels in plaintext. The flakiness table (which needs test titles) intentionally stays local-only.
• Graceful fallback: on any network failure the trend renders from the local history file, exactly as before.
• Per branch: keyed by project and branch, so a feature branch's runs never pollute main's trend line.

Off by default. Enabling it is the only thing that sends trend numbers to the server; the full inventory lives in the Failure Analysis page's "Data & network surface" table.

Prefer keeping even aggregate numbers off the server? Set historyFile to a project-relative path and cache it between runs so history accumulates across CI jobs.

// playwright.config.tsreporter: [['@reportforge/playwright-pdf', {  template: 'detailed',  historyFile: '.reportforge/history.json',}]]

# .github/workflows/test.yml- uses: actions/cache@v4  with:    path: .reportforge/history.json    key: reportforge-history-${{ github.ref }}    # Omit restore-keys; cross-branch fallback causes misleading sparklines on PRs- run: npx playwright test  env:    RF_LICENSE_KEY: ${{ secrets.RF_LICENSE_KEY }}    PUPPETEER_EXECUTABLE_PATH: /usr/bin/google-chrome-stable

Using github.ref as the cache key keeps history separate per branch. Drop restore-keys entirely on PR branches; cross-branch history makes the sparkline misleading.

Bitbucket Pipelines note: Bitbucket caches are immutable once written, so an evolving history file freezes at its first version — you would need rotating cache names to work around it. On Bitbucket, prefer remoteHistory: true above.

In a monorepo with multiple Playwright configs, give each config its own historyFile so runs from different packages do not overwrite each other.

// packages/frontend/playwright.config.tsreporter: [['@reportforge/playwright-pdf', {  historyFile: '.reportforge/frontend-history.json',  outputFile: 'reports/{date}-frontend.pdf',}]]// packages/api/playwright.config.tsreporter: [['@reportforge/playwright-pdf', {  historyFile: '.reportforge/api-history.json',  outputFile: 'reports/{date}-api.pdf',}]]

The detailed template shows a Top flaky tests table below the pass-rate trend card. Each row shows the test name, flake rate, how many runs it was flaky in, and a dot sparkline across stored history (amber dot = flaky in that run, grey = clean).

reporter: [['@reportforge/playwright-pdf', {  template: 'detailed',  flakinessTopN: 5,        // default: show top 5 flakiest tests                           // set 0 to hide the table entirely  quarantineThreshold: 40, // default: flag tests at 40%+ flake rate                           // set 0 to turn flagging off}]]

Rows also carry quarantine flags and a streak column: a test whose flake rate reaches quarantineThreshold across 3 or more recorded runs gets a QUARANTINE chip, and a callout above the table counts every candidate (even ones outside the top-N cut). The streak column shows how many consecutive latest runs the test has been flaky, so a long-running offender reads differently from a one-off blip.

The flakiness table is gated on showTrend: true (the default) and only appears when history contains at least one qualifying run. Runs recorded before this feature was introduced are excluded from the flake-rate denominator; the table fills in correctly as newer runs accumulate. No history migration needed.

A test is counted as flaky when Playwright marks it flaky, i.e. it failed on at least one retry attempt but ultimately passed.

Every report includes a Since Last Run section (sections.runDiff, on by default in all three templates) comparing the current run against the previous run on the same branch: new failures, newly flaky tests, fixed tests, and tests still failing. Tests that simply did not run this time are called out separately and never counted as fixed.

The diff reads the local history file only, the same one flakinessTopN uses; remoteHistory feeds the trend chart above, not the diff. The first run after upgrading has no comparable prior run to diff against, so the section starts appearing from the second run onward.

The same new-failures / fixed / still-failing summary also rides along in Slack, Teams, Discord, and email notifications.