Skip to content

Aim a test set at staging or production

A test whose steps carry https://staging.example.com/sites/42 is a test that has to be edited the day you want the same check against production. An environment removes that: it gives a place a name and a base URL, and a step written as /sites/42 resolves against whichever environment the run picks.

A project can have as many as you like. Environments are not a paid feature — unlimited on every plan, Free included. They are not what costs us money; runs are.

FieldWhat it does
NameWhat a run asks for: staging, production. Matched ignoring case
Base URLWhat a step’s /path is joined to
DefaultThe one a run uses when it names none
Navigate allowlistHosts this environment’s runs may reach
Signed-in sessionThe stored cookies for this place

The last two are per environment on purpose. A staging session cookie is worthless on production, and allowing a staging host must not quietly allow it for a production run.

They live only on the environment. A project has no copy of its own, and changing which environment is the default moves nothing: each keeps its own allowlist and session. (A copy kept on the project once followed “the default” around, and a rename carried a stored session into an environment that was meant to run signed out.)

A step target that starts with / is joined to the base URL. Anything else — a full https://… URL — is left exactly as written, so nothing you have today changes.

The join keeps the base URL’s own path:

Base URLStep targetRuns against
https://example.com/sites/42https://example.com/sites/42
https://example.com/shop/sites/42https://example.com/shop/sites/42
https://example.com/shophttps://other.test/xhttps://other.test/x

One exception worth knowing: verify_url takes a fragment to match, not a path to resolve. verify_url: /pricing means “the URL must contain /pricing”, which is already true on every environment — so it is left alone rather than joined, because joining it would quietly turn that loose check into an exact-match assertion.

The safety property: a stale URL fails loudly

Section titled “The safety property: a stale URL fails loudly”

This is the real reason environments are worth having, not a side effect.

Because the navigate allowlist belongs to the environment, a step that still hardcodes https://staging.example.com/... cannot quietly run during a production run. The production environment’s allowlist does not cover the staging host, so the step fails and names the host it tried to reach.

Without environments, that same test would have run green against staging while the report said “production” — the worst possible outcome, because you would trust it.

The allowlist for API request steps belongs to the environment too, so the property above covers a call_api step as well: a request pointed at your staging API cannot run during a production run.

The two moments are checked against different lists, on purpose:

  • When you save a request, the host must be permitted by at least one of the project’s environments. A request written for staging alone is not refused because production has never heard of that host.
  • When a run happens, the host must be permitted by the one environment that run resolved. This is the check that matters, and the one that did not exist before: the host was verified when the request was saved and never again, so narrowing an allowlist left every request authored earlier still callable.

A request the run’s environment does not permit is not sent to the runner at all — the URL stays on our side — and the step fails naming both the host and the environment, so it reads as a configuration problem rather than a product failure.

  1. Add an environment with a name (staging) and a base URL (https://staging.example.com) — from a coding agent with create_environment, or POST /api/v1/cli/projects/{id}/environments. The first one, Default, already exists: give it a base URL with update_environment.

  2. The base URL’s host is added to that environment’s allowlist for you. An environment cannot point somewhere its own runs are forbidden to go.

  3. Write steps with relative paths from then on: /, /pricing, /sites/42.

  4. Add production the same way when you are ready, and mark whichever one should be the default (make_default).

A run names an environment, or takes the project’s default. The run records the name it was asked for, and each test in the run records the environment it actually resolved — its name and base URL — so a screenshot can always say which place it came from.

From the API and a coding agent, the environment is named, never addressed by id:

Terminal window
curl -X POST https://api.marriska.com/api/v1/cli/cloud-runs \
-H "Authorization: Bearer $MARRISKA_API_KEY" \
-H "Content-Type: application/json" \
-d '{"test_sets": ["Regression"], "environment": "staging"}'

A run on your own machine names it the same way:

Terminal window
marriska-runner run --test-set Regression --environment staging

Both paths resolve the environment on the server and both refuse a name the project does not have — a suite aimed at production never quietly runs against staging because somebody mistyped it.

test_sets names sets the same way, so a CI workflow reads as what it runs and survives a set being recreated — which changes the id and keeps the name. A name used by two projects is refused, listing the candidates, and project narrows it. Nothing is resolved by picking the first match.

A test set is environment-agnostic: the run picks. If a set contains tests from several projects, the name is resolved per project, so staging means “each project’s staging”. A project that has no environment by that name refuses the run and says which project — it does not fall back to someone’s default.

Refused in two cases, both deliberate:

  • A schedule fires against its name. Retarget or delete the schedule first. A nightly suite is never moved to another environment for you — that is how a destructive suite ends up running against production and is discovered by its damage.
  • It is the project’s default and other environments exist. Make another one the default first, so nothing resolves to a surprise.