Architecture

Canon Ops: the design in ten minutes

An operator for restaurant order flow on a delivery-integration platform, shaped after the public API and data feed of a Gulf delivery-integration platform. It finds what goes wrong between the delivery apps and the kitchen, fixes what a written rulebook allows, asks a person for the rest, and grades itself. The full design has the data plane, the state machines, the scale and cost arithmetic, the rollout gates and the failure modes. The running copy is the reference implementation on a synthetic world: invented restaurants, labelled as such. Everything about the platform that is not in its public documentation is marked "(assumption)".

1. The problem, in one evening

Friday, 20:40, Riyadh. An order is placed on a delivery app. The app hands it to the platform, which relays it to the restaurant's till. The till's login token expired an hour ago, so the order sits accepted on the app and never reaches the kitchen. At 350,000 orders a day, a 99.95% injection rate still leaves about 175 orders a day like this one, before expiring tokens, menus out of sync, silent printers, overloaded kitchens and delivery apps going dark. Today a person in an ops room catches each one, eventually.

Canon sees the order at the 90-second mark, refreshes the token, re-sends the waiting orders, and writes down what it saw, the definition it applied, the exact call it made and how to undo it. Mean time from first seen stuck to the re-send in the replayed week: 63 s. A fix that touches guests, such as taking a store offline on every app, waits for a person, with the money at stake shown. Refunds, prices and contracts it never touches.

2. What runs

flowchart TB
    FEED["Order feed, POS logs, menus, dispatch"] --> DEF["Definitions: versioned YAML"]
    DEF --> DET["Detectors on the stream"]
    DET --> CTRL["Controller: policy, budgets, never-list"]
    CTRL --> ADPT["Adapters on the platform's verbs"] --> GW["Key gateway"] --> API["Platform APIs"]
    CTRL --> Q["Queue for people"]
    CTRL --> LED["Append-only ledger"] --> REC["Accuracy against ground truth"]

• Definitions say what is true: what a stuck order, an overloaded store or an active client is.
• Detectors evaluate them on the live events and open one incident per problem.
• The controller holds the rulebook and decides what may be done. A model is asked only where the runbook leaves a real choice, and it can only propose.
• Adapters call the platform's own verbs (working mode, store status, item availability, menu push, the POS relay).
• The ledger keeps every step with evidence, version, call and undo. Accuracy scores it against ground truth.

3. Taxonomy: the definitions are code

A definition is one file with an owner, a version, a dated change note, SQL and tests. The same file is compiled twice: to SQL for the questions people ask, and to a streaming check for real time. Change the threshold in stuck_order from 90 to 120 seconds and the alarms, the reports and the answers all change together, with lineage, and old answers stay reproducible by version.

Where a term has more than one valid meaning, the system keeps all of them. "Active client" has an ops version, a growth version and a finance version. Ask the question and you get three numbers side by side with the merchants in each gap, not one number that one department will dispute.

Every term the operator runs on, with its owner, inputs, first action and tier, is in the taxonomy table of the full design.

4. Decision model: who holds the authority

The model never holds authority. A separate controller owns the policy file and checks every proposal, in order: known action, not on the never-list, on the kind's runbook, arguments valid, conditions met, evidence fresh, no kill switch, cooldown clear, budget left, then the tier.

Budgets are split so back-office checks cannot starve the dinner service. Every call carries an idempotency key, and the platform credential sits behind a gateway that passes only the policy's verbs. A person can approve, reject or roll back any row; "approve and remember" adds a scoped rule and never edits the policy itself.

5. Measured, then rolled out

Shadow mode first: the operator logs what it would do and sends nothing, for two weeks in one market. Precision and recall are measured per detector per day against reviewed labels, and a kind goes live only after its reviewed shadow precision is 95% or better, with a kill switch tested live. Then more kinds, one at a time.

What the reference implementation shows today, on a week its definitions were never tuned on:

The split per detector, with the weakest one named, is on the Accuracy screen. Scores against a synthetic answer key are not measurements of a real platform; they show that the measuring works.

6. Honest limits

• Synthetic data throughout. The first real step is a read-only reader on the platform's order feed, in shadow mode.
• The re-send runs through the platform's internal POS relay. Whether a partner can trigger it is an assumption.
• The model tonight was a small local one, median call 5.6 s. The design does not depend on which model.
• The controller is a separate class in the same process. Separate processes and credentials are production work.
• The full list is in known gaps and in section 8 of the full design.