Staging vs Production CORS Parity Checks

Verified in staging, blocked in production:

Access to fetch at 'https://api.northwind.example/v2/orders' from origin
'https://dash.northwind.example' has been blocked by CORS policy: The
'Access-Control-Allow-Origin' header has a value
'https://dash-staging.northwind.example' that is not equal to the supplied origin.

Root Cause

Production is answering with staging’s allowlist. The release was tested end to end against dash-staging.northwind.example, where every check passed, and the production environment was created by copying the staging environment’s variable set — including CORS_ALLOWED_ORIGINS. Nothing in the repository changed between the two, so no diff, review or test could have shown it: the divergence lives entirely in per-environment configuration.

This is the characteristic shape of a CORS environment failure. The application code is identical, the container digest is identical, the test suite is green, and the policy is still different, because a CORS policy is assembled from contributions made by the CDN zone, the WAF, the ingress controller, the application and the environment variables — every one of which is scoped per environment. A parity check is the answer: compare what the two environments actually return, declare which differences are legitimate, and treat the rest as drift. This page is part of CORS Testing & Regression Prevention, which covers the wider set of checks a policy needs.

Prerequisite State

Step 1 — Declare what may legitimately differ

Parity is not “the two responses are identical” — the hostnames must differ, and a short Access-Control-Max-Age in staging is a deliberate choice that makes policy changes take effect in a minute rather than ten. Write those exceptions down. A difference that is not in this file is drift, by definition.

# parity.yml
route: /v2/orders
method: PATCH
request_headers: authorization, content-type

environments:
  staging:
    api: https://api-staging.northwind.example
    frontend: https://dash-staging.northwind.example
  production:
    api: https://api.northwind.example
    frontend: https://dash.northwind.example

# Directives allowed to differ, with the expected value per environment.
allowed_differences:
  access-control-max-age:
    staging: "60"
    production: "600"

# Everything else in this list must be byte-identical after canonicalisation.
must_match:
  - access-control-allow-origin
  - access-control-allow-methods
  - access-control-allow-headers
  - access-control-allow-credentials
  - access-control-expose-headers
  - vary

Keeping the exception list explicit is what stops the check from decaying. The alternative — a growing pile of ignore rules added whenever the job goes red — ends with a check that compares nothing.

Step 2 — Capture and canonicalise both snapshots

Raw responses cannot be compared directly: the origin values legitimately differ, header names are case-insensitive, and list values reorder freely. Canonicalisation turns both responses into a form where a remaining difference is necessarily meaningful.

Canonicalising a policy snapshot before comparing environments Four stages run left to right: capture, substitute, canonicalise and compare. Beneath each stage the running example shows the Access-Control-Allow-Origin value turning from a live hostname into a token and finally into a parity verdict. Parity compares canonical forms, never the raw strings the servers returned capture read the live preflight headers substitute swap env hostnames for shared tokens canonicalise lowercase names, sort and trim lists compare diff the two canonical forms dash.northwind.example {FRONTEND} {frontend} equal — in parity The running example is one header value; the same four stages apply to the whole response 
# parity_check.py — python parity_check.py parity.yml
import sys
import requests
import yaml

TOKENS = ("api", "frontend")


def capture(cfg, env):
    e = cfg["environments"][env]
    res = requests.options(
        e["api"] + cfg["route"],
        headers={
            "Origin": e["frontend"],
            "Access-Control-Request-Method": cfg["method"],
            "Access-Control-Request-Headers": cfg["request_headers"],
        },
        allow_redirects=False,
        timeout=10,
    )
    return e, res


def canonical(env_cfg, res):
    """Lowercase names, tokenise environment hostnames, sort list values."""
    out = {"__status__": str(res.status_code)}
    for name, value in res.headers.items():
        name = name.lower()
        if not (name.startswith("access-control-") or name == "vary"):
            continue
        for token in TOKENS:
            value = value.replace(env_cfg[token], "{" + token + "}")
        parts = sorted(p.strip().lower() for p in value.split(",") if p.strip())
        out[name] = ", ".join(parts)
    return out

Tokenising before splitting matters: https://api.northwind.example contains no comma, but an Access-Control-Expose-Headers list does, and sorting a list that still holds a raw hostname would produce a different order in each environment purely because the hostnames sort differently.

Step 3 — Diff, and classify every remaining difference

With both sides canonical, the comparison is small enough to fit in one function, and every finding it reports is real.

def compare(cfg, left, right):
    findings = []
    allowed = cfg.get("allowed_differences", {})
    keys = set(cfg["must_match"]) | {"__status__"}

    for key in sorted(keys):
        a, b = left.get(key), right.get(key)
        if a == b:
            continue
        if key in allowed:
            continue                       # declared, and checked separately below
        if a is None or b is None:
            findings.append(f"DRIFT  {key}: staging={a!r} production={b!r} (present in one only)")
        else:
            findings.append(f"DRIFT  {key}: staging={a!r} production={b!r}")

    for key, expected in allowed.items():
        for env, snap in (("staging", left), ("production", right)):
            got = snap.get(key)
            if got != expected[env]:
                findings.append(f"DRIFT  {key} in {env}: expected {expected[env]!r}, got {got!r}")
    return findings


if __name__ == "__main__":
    cfg = yaml.safe_load(open(sys.argv[1]))
    s_cfg, s_res = capture(cfg, "staging")
    p_cfg, p_res = capture(cfg, "production")
    problems = compare(cfg, canonical(s_cfg, s_res), canonical(p_cfg, p_res))
    for line in problems:
        print(line)
    print(f"{len(problems)} drift finding(s)")
    sys.exit(1 if problems else 0)

Run against the environments that produced the console error at the top of this page, it prints three findings — and the interesting part is that only one of them was the cause of the reported failure. The other two were latent.

The effective policy in each environment, side by side Two panels list the same five CORS directives for staging and production. The allowed origin, the max age and the Vary header differ; the methods and request headers match. Two of the three differences are drift and one is declared. staging — api-staging.northwind.example access-control-allow-origin https://dash-staging.northwind.example access-control-allow-methods delete, get, patch, post access-control-allow-headers authorization, content-type access-control-max-age 60 (declared difference) vary origin every check passed here before the release production — api.northwind.example access-control-allow-origin https://dash-staging.northwind.example access-control-allow-methods delete, get, patch, post access-control-allow-headers authorization, content-type access-control-max-age 600 (declared difference) vary absent two drift findings, one of them silent so far Same image, same commit, same tests — and a different policy in each environment

The missing Vary: Origin in production is the latent one. Nothing fails today because production currently answers with a single origin value, but the moment the allowlist is corrected to hold two origins, a shared cache will start handing one tenant’s grant to the other. The mechanics of that failure are set out in How to Fix Missing Vary: Origin Header Breaking CORS Cache Segmentation.

Step 4 — Trace each finding to the layer that produced it

A parity report tells you that the environments differ, not where. Because the policy is contributed to by several layers, the next step is to probe inward, one hop at a time, and find the layer at which the two environments stop agreeing.

Which layer contributed the divergence Five layers from the CDN down to the environment variables are listed for staging and for production. The CDN rule, the ingress Vary rule and the environment variable differ; the WAF and the application middleware are identical in both. staging production CDN zone no CORS rule configured on the zone CDN zone a transform rule adds a grant of its own WAF passes response headers through WAF passes response headers through Ingress adds Vary: Origin on every response Ingress the Vary annotation was never applied Application same image digest, same middleware Application same image digest, same middleware Environment variables CORS_ALLOWED_ORIGINS = dash-staging… Environment variables copied verbatim from staging, never edited Three layers diverge; only one of them lives in the application at all

The practical technique is to send the same preflight to progressively less public addresses and watch where the answer changes — the public hostname, then the load balancer’s own address, then the pod or instance directly. The first hop whose answer differs from the hop outside it is the layer that modified the response. That method is developed further in Troubleshooting CORS at the Proxy Layer.

# Same preflight, three vantage points — the first difference names the culprit
for target in https://api.northwind.example \
              https://lb-internal.northwind.example \
              http://10.42.7.19:3000 ; do
  printf '\n== %s\n' "$target"
  curl -sS --max-time 10 -k -o /dev/null -D - -X OPTIONS "$target/v2/orders" \
    -H 'Origin: https://dash.northwind.example' \
    -H 'Access-Control-Request-Method: PATCH' \
  | tr -d '\r' | grep -iE '^(access-control|vary):'
done

Step 5 — Run it on a schedule, not only at release time

Configuration drift does not wait for a deploy. A console edit, an expired rule, a Terraform plan applied to one workspace and not the other — all of these happen between releases, so the check has to run between releases too.

# k8s/parity-cronjob.yaml — every 15 minutes, from inside the cluster
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cors-parity
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: parity
              image: ghcr.io/northwind/cors-parity:1.4.0
              args: ["python", "parity_check.py", "/etc/parity/parity.yml"]
              volumeMounts:
                - name: config
                  mountPath: /etc/parity
          volumes:
            - name: config
              configMap:
                name: cors-parity-config

backoffLimit: 0 and concurrencyPolicy: Forbid are both deliberate. Retrying a parity failure would turn a real, persistent divergence into a job that eventually succeeds by accident, and overlapping runs would double the alert volume during an outage without adding information.

Verification

Security Boundary Note

Drift is directional, and the direction decides the severity. A production policy that is narrower than staging breaks a feature; a production policy that is wider is a security finding, because someone granted cross-origin access that no review approved. Treat an extra origin, an extra method, an Access-Control-Allow-Credentials that only production carries, or a wildcard that appears in one environment as an incident rather than a configuration nit.

The reverse temptation is just as dangerous. When the parity check goes red, the fastest way to make it green is to widen staging until it matches production, or to add the production hostname to the staging allowlist. Both make the two environments agree while making the policy worse. Fix the environment that is wrong, and if you genuinely cannot tell which one that is, the policy was never written down clearly enough — start from Wildcard CORS Risks and Safe Origin Allowlisting and decide the intended policy first.

Common Mistakes

Issue Technical impact Mitigation
Comparing raw responses instead of canonical forms Header case and list ordering produce constant false findings until the team mutes the job Lowercase names, tokenise hostnames and sort list values before diffing
Treating every difference as acceptable because “staging is different anyway” Real drift hides among the legitimate differences and reaches production unnoticed Declare the allowed differences in a file and fail on anything not listed
Probing an internal address for one environment and the public hostname for the other The comparison silently excludes the CDN and WAF from one side, so a whole layer goes unchecked Always probe both environments at the same depth, starting from the public hostname
Fixing drift by widening the narrower environment The two environments agree, and the policy is now more permissive than anyone approved Change the environment that diverges from the intended policy, never the one that matches it

FAQ

Which differences between staging and production are legitimate?

Hostnames, and a deliberately shorter Access-Control-Max-Age in staging so policy changes take effect quickly. Everything else should match: the same methods, the same request headers, the same exposed headers, the same credentials flag and the same Vary. Write the legitimate differences down as a declaration file, because a difference that is not declared is drift by definition — and the declaration is also the document that tells a future reader what the policy was meant to be.

Should a parity check block a production release?

It should block the release when it finds drift that widens the policy — an extra origin, an extra method, a credentials flag that only production carries — because those are security changes nobody reviewed. Drift that narrows production relative to staging should also block, since it means the feature verified in staging will not work for users. In both cases the fix is a configuration change, not an exception in the check. Wiring that gate into the pipeline is covered in Catching CORS Regressions in CI Pipelines.

Why does the same container image produce different CORS headers in two environments?

Because the image is only one of the contributors. The CDN zone, the WAF rule set, the ingress controller annotation and the environment variables are all per-environment, and any of them can add, rewrite or remove a CORS header. Probe each layer directly, from the public hostname inward, to find which one is responsible before changing anything in the application — a difference that appears at the edge and disappears one hop in was never the application’s doing.