L6 AI Operations Center#

Path: /aiops

The L6 center manages the full lifecycle of live AI applications: runtime monitoring, quality evaluation, business value, cost, optimization loop, A/B experiments and canary releases, and operations reports. The lifecycle doesn't end at L4 delivery — L1→L2→L3→L4→L6→optimization→re-assessment forms a continuous value loop; see the Business Flow.

Menu Structure (9 items)#

Menu	Path	Description
Operations dashboard	/aiops/overview	App health distribution/feedback/proposals/eval scores/cost/health score/running experiments
AI Applications	/aiops/applications	App registry (status/environment/agent/model/version/health/lifecycle stage)
Evaluation	/aiops/evaluation	App-level evaluation runs (reuses the platform EvalEngine)
Monitoring	/aiops/monitoring	Requests/success rate/failures/latency/tokens/cost/active users
Feedback	/aiops/feedback	👍👎/ratings/comments/corrections/issue types/severity
Cost & Usage	/aiops/cost	Daily/per-agent cost breakdown (in the user's currency)
Business Value	/aiops/value	Value period entry/trends/ROI actual vs predicted
Optimization	/aiops/optimization	AI proposals → approval → validation → comparison → adopt/rollback
Operations reports	/aiops/reports	Periodic aggregate reports + AI operations conclusions

App Registration & Versions#

• Auto registration: when a delivery project completes (ProjectCompleted event), an app is automatically created in the registry (with AI app metadata = running; otherwise = draft pending confirmation)
• Version management: each app has a version sequence (v1.0/v1.1...), supporting manual creation/activation; versions are ordered by internal sequence number; display numbers are identifiers only
• Version comparison: in the app detail Versions tab, compare two versions' eval scores and feedback (mutually exclusive validity windows to avoid confusion)
• Versions in experiments: versions in an A/B experiment show a golden "Canary N%" badge and cannot be manually activated — end the experiment first (or wait for auto-rollback); see A/B Experiments & Canary Releases

Optimization Loop#

• No direct go-live after approval: an approved proposal creates a pending-validation version and triggers automatic re-evaluation; adopt or roll back only after manual comparison
• Adopt: the new version goes live (the old one marked superseded), the app version updates
• Rollback: the new version is marked rolled back and the app keeps the original version (rollback = reactivating a historical version, not deletion)
• Re-evaluation comparison summary: after automatic re-evaluation completes, the proposal card shows a summary tag group (eval score changes + feedback changes; improvements in green, deteriorations in red; fewer negative reviews counts as an improvement)

Business Value & ROI#

• Value entry: enter business metrics per period (month) (baseline/target/actual), value created, cost, payback period
• ROI actual vs predicted: apps linked to an L2 decision scenario can compare predicted vs actual ROI deviation (percentage points/relative deviation + cause diagnosis); when deviation is significantly negative, generate an optimization proposal in one click
• Currency: value and cost are displayed in the user's currency setting (internally stored in USD)

Daily AI Operations Brief#

• Push time: daily at 09:00 (UTC 01:00); re-pushed automatically after backend restart if the time has passed
• Recipients: tenant owner/admin (content is the whole tenant's operations aggregate; each admin receives an identical copy)
• Content: total apps/health distribution/pending feedback/pending proposals/cost/success rate + abnormal apps/high-severity feedback/pending proposal highlights + LLM polish
• L1 re-assessment suggestion (L6→L1 flywheel): apps whose health status is critical (manually flagged in the app registry — the system never sets apps to critical automatically) or whose ROI relative deviation ≤ -30% are listed in the "🔄 Suggest initiating an L1 readiness re-assessment" section; L1 re-assessment is an enterprise-level manual process — the brief never creates assessment records automatically — go to L1 "Readiness Assessment" and initiate a re-assessment with current operational data (success rate/cost/user feedback) as evidence
• Dated title: Tenant AI Operations Brief · YYYY-MM-DD; each notification shows its creation time
• No duplicates: pushed at most once per tenant per day (three-layer idempotency: Redis lock + database unique constraint); backend restarts or multi-instance deployments never produce duplicate briefs

A/B Experiments & Canary Releases#

Entry: AI Applications → app detail → "Experiments" tab. Validate a challenger version against the baseline (current live version) with real traffic, and decide adopt/rollback with data.

Creating an Experiment#

1. Click "Create Experiment" (only one running experiment per app at a time; at least 2 versions required)
2. Fill in the form:
   • Experiment name: e.g. v1.0 → v1.1 canary
   • Baseline version (Champion): only the currently active version
   • Challenger version (Challenger): only versions in draft/pending-validation/validated states with a configured model
   • Traffic ratio (%): 1-100, default 10
3. After creation the challenger enters canary testing; the version list shows the golden "Canary N%" badge; the operations dashboard's running experiment count updates

Traffic Assignment (Session Stickiness)#

• Deterministic per-session splitting: repeated calls in the same session always land on the same version, never jumping between the two; the split ratio approximates the configured value
• Sessions assigned to the challenger are served by the challenger's configured model; the rest go to the baseline; the challenger must have a configured model (versions without a model never appear in the candidate list and can't join experiments), otherwise the split has no real effect

Viewing Metrics#

Click "View Metrics" in the experiment list to open the comparison drawer:

• Two-version comparison table: version/model/executions/success rate/average latency/cost
• Δ change tags: success-rate up in green; average latency/cost down in green (red otherwise)
• Experiment-period feedback: user feedback since the experiment started, grouped by severity

Metrics come from the execution ledger (real execution records of every AI call), accumulated from the experiment start.

Ending an Experiment (Manual Decision)#

Select the winner in the metrics drawer and confirm:

Winner	Result
Challenger	Challenger goes live (app's current version and model update); the baseline is marked superseded
Baseline	Challenger retires; traffic immediately returns 100% to the baseline

The experiment record is marked completed; start/end times remain in the list; the winner and end reason are archived in the record (not shown in the current UI; queryable via API).

Auto-rollback (Watchdog)#

The "Auto-rollback" config card at the bottom of the Experiments tab:

Config	Default	Description
Enable auto-rollback	Off	App-level switch; takes effect only when explicitly enabled
Health threshold	60	Triggered when the app health score drops below
Minimum sample size	50	Challenger samples must reach this before triggering, preventing small-sample misjudgment

• When enabled, the watchdog patrols canary traffic every 15 minutes; trigger conditions (either one, with sufficient samples): app health score < threshold, or challenger success rate < 80%
• The watchdog is enabled only in production by default — test/demo environments never patrol or auto-rollback even with the switch on
• On trigger: challenger rolls back immediately (traffic 100% back to baseline) → experiment marked completed (winner = baseline, reason = auto-rollback) → high-priority notification "Canary Auto-rollback · App name" sent to the app owner and owner/admin with real-time push
• Concurrency guard: row-level lock + state guard; only one side wins when racing with manual experiment end — no double rollback

Data Semantics#

Experiment metrics, monitoring and cost statistics are all backed by the execution ledger (ai_executions): every AI call is recorded (agent/model/tokens/cost/success). Experiment metrics come only from the ledger (accumulated from experiment start); monitoring and cost fall back to chat-record aggregation when ledger data is absent for a period.