Testing Rule Sets

Built-in test cases let you pin the expected behavior of a rule set and prove it still holds after every edit. Tests run synchronously in a sandbox — no real side effects, no metering, nothing in run history — and can gate a release so unreviewed changes never ship.

Tests and verification are complements A test proves an input produces the outcome you expect. The Verify tab catches what a passing test cannot see — a rule that never fires, two rules writing the same key, a context key nothing supplies — without running anything. Use both.

Test cases

A test case belongs to one rule set and captures:

• Input context — the same typed {DataType, Value} entries a real run receives.
• Assertions — expected outputs, written as ordinary condition clauses evaluated against the final context. You get Equals, Contains, GreaterThan, In, Between, regex and null checks for free. (AI-prompt operators are not allowed as assertions — assertions must be deterministic.)
• Expectations — whether the run should have errors or warnings, and which rules should or should not have matched.
• Mocks — canned outputs for side-effecting actions and AI conditions (see below).
• Run as of — an optional simulated date and time for the run (see below).
• Tags — lightweight grouping/filtering. There is no separate "suite" entity; the suite is simply every test on the rule set.

Capture a known-good run From the runner you can Save as test: run once in the sandbox, tick the context keys you care about, and the assertions are pre-filled with Equals and the actual values. That is the fastest way to lock in a regression test.

Sandbox semantics

Tests always execute the current draft (the live working copy) — that is what a release candidate is. Execution is fully sandboxed:

Nothing a test does appears in run history or logs, and your monthly run count is never touched. Test execution is free on every plan; only the number of test cases per rule set is plan-limited (Free 5, Pro 50, Team 200, Enterprise unlimited).

Mocks

Three kinds of mock make a run deterministic:

• Action — matched by the action's name; its outputs are merged into the context in place of the real call.
• AI condition — matched by the clause's context key; supplies the match result so no live, billed AI call is made.
• Human intervention — matched by the action's name; supplies the outputs the human would have provided.

Unmocked non-determinism fails fast An unmocked AI condition or human-intervention action fails the test with a clear error rather than hanging or making a live call. Opt into a real AI call per-test with Allow live AI (uses your organization's own provider keys).

Testing dated rules with “Run as of”

A rule can have an active date window, which makes it impossible to cover with an ordinary test once the date has passed — or before it arrives. Set Run as of on the test case to pretend the run happens at a specific date and time; every rule's effective/expiration window is judged against that instant instead of the real clock. Leave it blank (the default, and what every existing test does) to use the current date and time.

Write one test just inside the window and one just outside it, and the pair keeps proving the rule turns on and off when it should — forever, not just on the day you wrote them. A rule set invoked from within the run inherits the same simulated clock, so a nested rule set's dated rules resolve consistently with the outer one.

Reading a test result

A result shows each assertion's pass/fail against the final context, the run's errors and warnings, and which rules matched. Two extras help you see why a result came out the way it did:

• Inference trace — for a rule set in Inference mode, the result includes the full firing-by-firing trace. Test runs are always traced, so you get the timeline, each “why it matched” condition tree, the context deltas, and any derived facts without opting in.
• What if? — a panel seeded with the test's own input and mocks. Edit an input and Compare to run baseline vs. modified in the same sandbox and see which outcomes flip, without touching the saved test. It is the same what-if comparison offered in the runner.

The release gate

A project owner can turn on Require tests to pass (next to Require approval in project settings). When on, a rule set cannot be approved or published unless the latest full-suite run is green and its content hash matches the exact candidate being released. Editing the rule set after a green run invalidates it (“stale — re-run”), and a gate with zero tests defined is blocked rather than passing vacuously. The certifying run id is recorded on the release for the audit trail.

After a rule set is published Publishing finalizes one rule set. A project release is the layer above it — a numbered snapshot of every rule set, condition, and action set in the project at one moment, promoted stage by stage across your environments with its own approval gates.

Running tests from CI

Create a rule-set-scoped application API key and call the test endpoint with it. The call runs the whole suite and returns the run as JSON; fail the build when Failed > 0.

# PowerShell
$run = Invoke-RestMethod -Method Post `
  -Uri "https://<your-environment>/api/v1/rulesets/<ruleSetId>/test-runs" `
  -Headers @{ "X-API-KEY" = "$env:RULE_ENGINE_API_KEY" }
if ($run.Failed -gt 0) { throw "$($run.Failed) of $($run.Total) rule tests failed." }

# bash + jq
run=$(curl -s -X POST \
  -H "X-API-KEY: $RULE_ENGINE_API_KEY" \
  "https://<your-environment>/api/v1/rulesets/<ruleSetId>/test-runs")
failed=$(echo "$run" | jq '.Failed')
[ "$failed" -eq 0 ] || { echo "$run" | jq '.Results[] | select(.Passed==false)'; exit 1; }

In an Azure Pipelines Test stage, run one of the above as a script step before your deploy stage so a red suite blocks the deployment.

From an MCP agent

An MCP client whose credential can reach the rule set can call the built-in ruleset_run_tests tool with { "ruleSetId": "..." }. It runs the full suite in the sandbox and returns the pass/fail summary — letting an agent close the loop: edit rules, run tests, read the failures, fix, repeat.