Experiments & feature flags

A feature flag serves each user a variant — deterministically, in the SDK, with no per-user server round trip. An experiment is an analysis window on top of a flag: it compares treatment variants against a control on a metric you choose and tells you the chance each variant actually beats it. Everything is driven from the ah CLI; the dashboard renders the same numbers at /flags.

How assignment works

The SDK downloads the project's flag ruleset once (then caches it), and computes hash(flagKey + salt + anonId) locally. Same user, same variant — across sessions, launches, and offline stretches. Two consequences worth knowing: ramping a flag's traffic percentage up never reshuffles existing users (the variant hash is independent of the enrollment hash), and editing variant weights does reshuffle — which is why the CLI refuses weight edits while an experiment is live.

Users outside the traffic allocation — or when a flag is disabled, unknown, or the ruleset hasn't loaded yet — get undefined: your code default applies. That is the kill switch: ah flags disable <key> puts every client on the fallback within ~2 minutes.

Reading a flag

Web — after the tracker script loads:

await window.agenthog.flagsReady()               // optional: avoid the first-visit undefined window
const variant = window.agenthog.flag('checkout_cta')  // 'control' | 'b' | undefined
if (variant === 'b') renderNewCheckout()
else renderOldCheckout()                          // control AND fallback path

window.agenthog.flag('new_nav')                   // boolean flags → true | false | undefined

React Native / Expo:

const ah = useAgentHog()
await ah.flagsReady()
const variant = ah.flag('checkout_cta')

Capacitor:

await AgentHog.flagsReady()
const variant = AgentHog.flag('checkout_cta')

The first flag() read does two things automatically: it emits one $exposure event per flag per session, and it stamps $ff/<key>: <variant> onto every subsequent event. That is the entire data plumbing — you never hand-instrument which variant a user saw, and any metric splits by variant: ah events --by flag:checkout_cta, ah funnel signup --by flag:checkout_cta. For local testing, overrideFlag('checkout_cta', 'b') forces a variant without polluting experiment data.

The 24-hour loop

The workflow this is built around — an onboarding test, from idea to shipped winner, in a day. Config changes reach clients in ≤ 2 minutes and results are live SQL, so the loop is as fast as your traffic allows:

# day 0, morning — create the flag, open the experiment window
ah flags create onboarding_v2 --variants control:50,b:50
ah experiments start onboarding_v2 --metric onboarding_complete --min-days 1

# (wire agenthog.flag('onboarding_v2') into the app, deploy)

# day 0, any time — numbers are live from the first exposure; peeking is fine
ah experiments results onboarding_v2

# day 1, morning — ★ verdict, ship it
ah experiments stop onboarding_v2 --winner b
ah flags weights onboarding_v2 b:100

# later — retire: archive the flag, delete the dead code path
ah flags archive onboarding_v2

Iterate by windows, never by editing a running test: stop closes the analysis window, adjust the flag, and start opens a fresh one on the same flag. Data never mixes across windows; ah experiments results <key> --history lists past ones.

How results are computed

The joins are the honesty of the whole system, so they are worth stating exactly:

Stats are Bayesian: each variant gets a chance-to-beat-control and a 95% credible interval. Numbers render live and checking hourly is fine — but the ★ verdict only appears once every arm has ≥ 100 exposed users, the experiment's --min-days have elapsed, and an arm clears 95% chance-to-beat. Use --min-days 1 for same-session metrics like onboarding; keep the default 7 for anything with weekly rhythm (the output carries a day-of-week caveat below 7). ah experiments start also prints a sample-size hint from your last 30 days so you know whether the test is worth running before you run it.

Secondary and guardrail metrics

--secondary e1,e2 adds diagnostic metrics computed the same way. --guardrail app_error:up watches for regressions — the direction is the bad direction (errors:up, activation:down); results flag ⚠ when a treatment is ≥ 90% likely to have moved a guardrail the wrong way, even while the primary metric looks like a win.