What is Reality Verification? — Orbita Knowledge Base

Definition

Reality verification asks a strict question: “Did this behavior work in the same environment where operators depend on it?” If the answer is only “it compiled” or “curl returned 200,” verification is incomplete.

In Orbita, reality verification is hierarchical:

Real-device truth (highest confidence where relevant).
Browser automation for repeatable UI/system checks.
HTTP smoke for endpoint and chain sanity.
Build checks for syntax and bundle integrity.

This hierarchy prevents “works on my machine” claims from overriding operator evidence. When real-device behavior conflicts with synthetic tests, investigation continues until the gap is explained.

Verification is therefore an evidence practice, not a confidence feeling. Teams document what was tested, where it was tested, and why the result should be trusted for the target deployment context.

Reality verification also requires boundary awareness. A passing test in one universe (for example, local runtime with seeded data) does not certify another universe (production runtime with real operational rows). Cross-universe assumptions are a common source of false pass declarations.

Purpose

The purpose is to stop false confidence from entering production decisions. Operational systems fail most often at boundaries: session cookies, redirects, mobile scan timing, runtime configuration, and tenant-specific data states.

Protect operators from cert claims that ignore actual workflows.
Protect releases by enforcing runtime/source match and chain-level validation.
Protect governance by requiring evidence, not assumptions, in completion reports.

Reality verification does not reject automation. It places automation in the correct confidence layer and requires escalation when higher-layer evidence contradicts lower-layer assumptions.

This protects both users and developers. Users avoid unresolved defects hidden behind green badges, and developers avoid repeated incident cycles caused by incomplete closure standards.

This discipline is especially important for scanner and mobile workflows, where timing, focus behavior, and network conditions differ substantially from desktop test contexts.

Workflow

Declare universe: runtime, device class, data target, and scope.
Validate runtime identity: ensure deployed source and running service match expected revision.
Run chain smoke: confirm core path responds and dependencies are reachable.
Run scenario proof: execute operator-relevant flow (often UI/mobile specific).
Collect failure evidence if mismatch appears: logs, screenshots, network traces, state context.
Classify status: PASS, FAIL, or INVESTIGATE with explicit blocker and next action.

The critical point is closure discipline: a failed higher-confidence test cannot be closed by rerunning lower-confidence checks.

A complete verification receipt should therefore include environment, runtime target, scope, result class, and explicit next action when status is not pass. Without this, teams cannot distinguish unresolved risk from completed validation.

In cross-module changes, receipts should include upstream and downstream checks, because endpoint-level success can still mask chain breakage at handoff points.

Example

A picking flow smoke passes in desktop browser, but warehouse operators report mobile scan lockups. Reality verification marks status as investigate, gathers device evidence, and isolates whether the issue is input focus, camera decode cooldown, or session boundary behavior. The incident is not closed merely because endpoint smokes remain green.

Another case: localhost tests pass on port assumptions while actual runtime listens on a different configured port. Reality verification catches entry-url mismatch before declaring route regression fixed.

A third case appears in authentication flows: API checks return healthy status, but real browser sessions fail redirect loops due to cookie domain mismatch. Reality verification identifies this only when user-path testing is part of closure criteria.

These examples show a common lesson: verification confidence is proportional to scenario realism. The more the test resembles actual operator behavior and data conditions, the more reliable the certification claim.

FAQ

Is build success enough?: No. Build success proves compile integrity, not runtime behavior.
Are smokes still useful?: Yes. Smokes are essential but not final authority when real usage contradicts them.
Why require environment declarations?: To avoid cross-universe assumptions where LOCAL and ONLINE behavior are conflated.
Does reality verification slow delivery?: It adds discipline early and reduces expensive production rollback and forensic work later.
Can one passing test close all environments?: No. Each target environment requires explicit validation scope and result declaration.
What if real-device evidence conflicts with automation?: Status stays investigate or fail until the contradiction is resolved with traceable proof.

Misconceptions

“If smoke passes, user report is likely wrong.” False. User evidence may reveal boundary conditions not in smoke.

“Runtime mismatch is minor.” False. Source/runtime drift invalidates certification confidence.

“Evidence collection is optional.” False. Without evidence blocks, root-cause claims are speculative.

When It Matters

Reality verification matters for any change touching login, scan workflows, route gates, session handling, finance posting paths, and cross-module chain transitions. These are areas where synthetic confidence often hides operational defects.

It matters equally in incident response. Treating contradictory user evidence as “non-reproducible” without captured artifacts usually prolongs outages and shifts cost to frontline teams.

For release governance, reality verification is the difference between optimistic rollout and operationally defendable rollout.