Skip to content
Marriska Docs

Visual regression

Visual regression catches when something looks wrong, not just when something is wrong. A button moved 3 pixels off, a color swap on the primary CTA, the search bar disappearing on Firefox — these don’t fail a “find the button and click it” check, but they’re real bugs.

This page is about how Marriska does that check, why we chose that approach, and what it costs you.

The two ends of the spectrum

Section titled “The two ends of the spectrum”

Visual testing has historically pulled in two directions, and both hurt:

Pure pixel-diff (Percy, Chromatic-style classic)

Section titled “Pure pixel-diff (Percy, Chromatic-style classic)”

Compare every pixel to a baseline. If any meaningful number changed, fail the test.

Why this loses: browsers update their font renderers, antialiasing algorithms shift between Chromium versions, sub-pixel rounding disagreements between OSes — none of which are bugs, but all of which trip a strict pixel-diff. You end up either bumping baselines weekly or living with red builds you’ve trained yourself to ignore.

Pure LLM-vision compare (the “ask the model” approach)

Section titled “Pure LLM-vision compare (the “ask the model” approach)”

Send both screenshots to a vision LLM and ask “did anything important change?”

Why this is expensive: vision tokens cost real money, and the overwhelming majority of step screenshots haven’t changed at all between runs. Burning a vision-model call on every step of every test gets pricey at scale, and the latency adds up.

What we do instead

Section titled “What we do instead”

Three-stage cascade. Each stage answers if it can; otherwise it punts to the next:

1. Byte-identical short-circuit

Section titled “1. Byte-identical short-circuit”

If the new screenshot is exactly the same bytes as the baseline, pass without doing anything else. Common when nothing on the page actually changed.

2. Pixel-diff fast path

Section titled “2. Pixel-diff fast path”

Compute the percentage of pixels that differ. If it’s under the threshold (default 0.5%, configurable via VISUAL_PIXEL_THRESHOLD), pass. This catches the antialiasing-and-rendering-noise case without spending an LLM call.

3. Vision-LLM judgement

Section titled “3. Vision-LLM judgement”

If the diff exceeds the threshold, send both screenshots to the visual-comparison model with a tight system prompt. The prompt asks specifically for meaningful regressions — layout shifts, missing elements, color/style changes, broken components — and explicitly tells the model to ignore dynamic content (timestamps, badges, counters) and rendering noise.

The LLM responds with PASS: or FAIL: followed by a one-sentence explanation. That sentence becomes the analysis attached to the step result, so when something fails you see why, not just that it did.

The verdict is the reply’s first word, read through any markdown around it (**PASS**: … is a pass). A reply whose first word is neither — or that says PASSED — is no verdict.

If the comparison gets no verdict — the call fails (network, the provider’s error, no API key configured), or the model answers nothing or neither word — the step fails with the pixel diff percentage and says the AI comparison was unavailable on our side. Its origin is ours: not your site, and never a pass. It is not counted against your monthly AI screenshot checks.

What counts as a “step” for visual regression

Section titled “What counts as a “step” for visual regression”

Any step that uses the screenshot (or takeScreenshot) action with a baseline saved on the test runs through the visual pipeline. Other step types capture a screenshot too, but those are for the report view — they don’t get compared.

To opt a step into visual regression:

  1. Add a screenshot: step in your test.
  2. Run the test once with visual_mode=update_baseline — this saves the captured screenshot as the baseline.
  3. Subsequent runs in the default visual_mode=compare mode go through the cascade above.

A visual check compares against a baseline that belongs to a saved step, so it needs a saved test and a saved step. Run from the editor before a new screenshot step is saved, that step fails and asks you to save and run again. It never reports a pass for a check that compared nothing. If the check cannot run on our side, it fails and says the problem is ours, not your site’s.

Baselines are keyed by step ID + browser, not by step number. Reordering steps or inserting new ones doesn’t invalidate baselines — the IDs survive drag-and-drop and edits. In the editor a step keeps its ID even when you change what it does, so its baseline stays: if you point a screenshot step at something new, run it once with update_baseline.

Replacing a test’s steps through the API (PUT /cli/tests/{id}/steps, a coding agent’s update_test_steps) works by what each step does: a step sent unchanged — the same action, target and value — keeps its ID and its baseline; a step that was removed or changed is replaced, and its baseline goes with it. The reply says how many went (baselines_removed). See Step actions reference for the screenshot shape.

What you pay (and how to pay less)

Section titled “What you pay (and how to pay less)”

Only the LLM stage costs anything, so only it is metered: each vision verdict is one AI screenshot check against your monthly quota (20 on Free, 100 on Starter, 500 on Pro, 2,000 on Team, unlimited on Enterprise — see Plan tier limits). The byte and pixel stages are free and are not counted, so a suite whose pages are stable can run visual steps all month without spending a check.

Two more rules worth knowing:

  • Past the cap, the step fails on the pixel verdict and says monthly AI screenshot check limit reached (N/N). It does not quietly stop consulting the model — a visual check that silently degraded to “passed” would hide the regression you added it to catch.
  • BYOK checks are not counted. With your own provider key the call lands on your account at your rates, so it is not ours to meter.

A check is booked only after the model answers, so a provider outage costs you nothing.

If the diff threshold is too tight for your test, raise the visual_pixel_threshold (it’s a fraction between 0.0 and 1.0). A slightly higher threshold means more passes from the pixel stage without an LLM call.

What we don’t do (yet)

Section titled “What we don’t do (yet)”

  • Visual checks on a local agent: a screenshot step fails there, because the baseline lives in the cloud — see Cloud vs local.

  • Per-element baselines independently of the step they live on — baselines are step-scoped. If you want N elements compared independently, write N screenshot steps.

  • Auto-baseline-update on intentional changes: today, intentional changes need a manual update_baseline run.

  • Cross-browser baseline sharing: each browser stores its own baseline, because Chromium / Firefox / WebKit render real differences (especially in form controls and SVG).

Section titled “Related”