Observatory is not in the execution path
Agents call their tools and models directly. Observatory receives telemetry about that work. It does not run the mission or route its calls.
Tracston · Interactive lab
AI Flow & Performance Monitoring (AFPM) See how your AI missions run. Find where they go wrong.
Follow one request through agents, models, tools and approvals. Find what it cost, where it waited, which policy it met, and whether the result held up.
Four investigations inside: CostLatencyPolicyOutcome
No sign-up. Fictional missions run in your browser; nothing touches real systems.
The animation beside this text shows a fictional mission as a 3D topology: a person submits a request to a Coordinator Agent, which branches the work into Research, Pricing and Validation agents that call a model endpoint, a browser, a CRM and an inventory API. The Pricing Agent repeats the search and datasheet summaries the Web Research agent already ran, and every model call carries a large context, so the simulated cost keeps climbing. Imperium captures the Pricing Agent’s model request with its token usage and cost; Observatory correlates it with the earlier calls and traces the cost back to the repeated work. The same mission can be investigated step by step in the lab below, in the “Where did the AI budget go?” scenario, one of four investigations there: cost, latency, policy and outcome.
TRACSTON / AFPM LAB Interactive simulation
Investigate AI agents with Imperium + Observatory.
Choose an investigation
Simulation · fictional data
DelegationTool callModel callOutputPolicy decisionApprovalCorrelation (not a call)Failure / timeout
Mission metrics at the current moment
Simulated figures0 agents0 tool calls$0.000 model cost0 issues
Event list
The same events the graph draws, as text. Only events that have happened by the current time are listed. Keyboard: Space plays or pauses, comma and period step between events, F fits the graph, R resets the view, Esc clears the selection.
| Time | Event | Source | Actor | What | Status |
|---|
Imperium captures and governs.Observatory correlates and explains.
AFPM Library
Labs, cases and evidence, organized by incident.
A reference for investigating AI missions: standalone labs to run, case files that start from the business request, and the technical evidence behind each case.
Topic
01 · Interactive Labs
Four standalone investigations
Separate from the four scenarios in the main lab: each lab is a guided investigation on its own page. Watch the baseline, change one control, replay, and compare both runs.
-
Where Did the AI Budget Go?
Model spend climbs while the quotation stops moving.
-
The Slow Mission
An inventory tool times out and the whole quotation waits.
-
Denied. Still Returned.
The policy says DENY. The tool returns the fields anyway.
-
Mission Complete. Business Failed.
Every agent is green. The approver rejects the quotation.
02 · Case Files & Stories
From the business request to the evidence
Each case starts with what someone asked for, shows what went wrong, and names the evidence that explains it, and who supplied it.
- Fictional simulation
- An invented mission: the customers, people and figures are made up, and it runs in your browser.
- Illustrative scenario
- A hypothetical an article uses to frame a question. Nothing was measured.
- Documented real incident
- A real incident with its public sources linked. None is published in the library yet.
Lab caseFictional simulationCost
Where Did the AI Budget Go?
Model spend climbs while the quotation stops moving.
The request
“Prepare a quotation for Northwind Labs: 40 × XR-200 sensors, delivery in March, under our framework agreement. Submit the draft for approval.”
d.levi (Sales), identity as reported · mission MSN-30101
What went wrong
Research fetched the customer’s RFQ and framework agreement and summarized them for the quotation. Pricing was then told to read the same documents itself: it fetched them again and sent all 41.8k tokens to the model a second time. For those 3.4 s the quotation did not move from “Context ready”.
The evidence that explains it
- Imperium captured both model requests with their agent, mission, input tokens (41,800 each), estimated cost and the content hash of the documents they carried.
- Observatory correlated the matching hash, the timing (Research’s summary reached the Coordinator before Pricing was delegated) and the identical extraction into finding F-L01-1: $0.157 spent on facts the mission already had.
- Infrastructure alone two successful model API calls of similar size. Nothing says the second repeated the first.
What changed it
Replay with Research’s summary handed to Pricing: model cost $0.41 → $0.25, elapsed 23.6 s → 20.0 s, the same five validation checks pass and the quotation is approved.
Run Lab 01Inspect the evidenceMain lab: Where did the AI budget go?
Lab caseFictional simulationLatency
The Slow Mission
An inventory tool times out and the whole quotation waits.
The request
“Prepare a quotation for Northwind Labs: 40 × XR-200 sensors, delivery in March. Confirm stock before you submit.”
d.levi (Sales), identity as reported · mission MSN-30102
What went wrong
The inventory API stopped answering. Pricing sent the same request 5 times; the first 4 each timed out after 6.0 s, and between attempts Pricing asked the model what to do and was told to retry. Research had finished at 6.4 s; validation, the draft and the approval all waited for stock.
The evidence that explains it
- Imperium captured each inventory request with the agent that sent it, the mission, the attempt number, the idempotency key and the payload hash, plus the 4 re-plan model calls in between.
- Observatory correlated the attempts into one retry loop with an unchanged payload, and placed it on the mission’s critical path: 33.6 s between Research finishing and stock being confirmed.
- Infrastructure alone 4 × HTTP 504 from inventory-api, with no caller, no mission and no sign of what was waiting.
What changed it
Replay with one attempt and a permitted fallback: the live read-only replica meets the stock requirement (48.8 s → 20.2 s, approved). The 26-hour stock cache is faster still (16.0 s) but fails the live-stock requirement, so the draft is blocked.
Lab caseFictional simulationPolicy
Denied. Still Returned.
The policy says DENY. The tool returns the fields anyway.
The request
“Prepare a renewal quotation for Northwind Labs: renew the XR-200 sensor contract for another year at the framework terms. Submit the draft for approval.”
svc-sales-automation, shared service account · initiator unverified
What went wrong
A CRM Lookup child agent asks for the customer record including tax_id and credit_limit. Imperium records the policy decision DENY: those fields are restricted to finance roles. But the child agent’s connector calls the CRM directly, not through the enforcing gateway. No enforcement is recorded, the CRM answers 200 OK, and the response carries both restricted fields.
The evidence that explains it
- Imperium governed and captured the policy decision (DENY, policy restricted-customer-fields), the absence of an enforcement record, and the tool response’s field list at the agent’s connector.
- Observatory correlated decision, missing enforcement and response into one finding: denied fields were returned. It keeps the three apart: a decision is not proof that anything was stopped.
- Infrastructure alone GET /customers/C-4471 → 200 from the service account. A normal read.
What changed it
Replay with the denial enforced at the tool boundary: the connector routes through the enforcing gateway, which removes both fields and records it. Restricted fields returned: 2 → 0; the renewal is approved in both runs. The person behind the shared account stays unverified in both: enforcement does not answer who asked.
Lab caseFictional simulationOutcome
Mission Complete. Business Failed.
Every agent is green. The approver rejects the quotation.
The request
“Prepare a quotation for Northwind Labs: 40 × XR-200 sensors, delivery in March. Submit the draft for approval.”
d.levi (Sales), identity as reported
What went wrong
Every agent finishes and every service answers 200 OK. Validation passes, but it checks schema, totals, currency and terms, not inventory confirmation. The stock figure came from a 26-hour-old cache with no reservation. The approver rejects the draft: “No stock confirmation for XR-200. Cannot send.”
The evidence that explains it
- Imperium captured what validation checked and what it skipped, the inventory tool’s actual answer (cached snapshot, no reservation), and the approver’s decision.
- Observatory correlated the skipped check, the cached answer and the rejection into two findings: a validation coverage gap, and a business outcome not achieved despite a completed mission.
- Infrastructure alone all services healthy. Nothing to report.
What changed it
Replay with inventory confirmation required before submission: validation fails the first draft, Pricing confirms live stock and reserves 40 units, and the approver approves. It takes longer (13.5 s → 19.3 s) and delivers a quotation the business can send.
StoryIllustrative scenarioCost
The rebuild nobody priced
A deployment succeeds. The failed first attempt and the rebuild never show up in what it cost.
The request
“Deploy the service and report back when the checks pass.”
From the Insights article “The Agent Finished the Job. Nobody Knows What It Cost.”
What went wrong
The deployment succeeds and every check passes. One line in the summary says the first virtual machine failed, so the agent deleted it and built another. Rebuilding may have been the right call. But nobody can say what the failed attempt, the rework and the waiting added to a result reported as a success.
The evidence that explains it
- What has to be captured each attempt, tool action and model call, tied to the mission and the agent, with elapsed time, resource use and the approvals the deletion needed, and whether they were enforced.
- What has to be correlated necessary recovery versus avoidable rework, and the cost of failure inside a successful outcome. Where coverage is incomplete, the gap should show.
The question it leaves
No figures: the article frames the question. Lab 01 and Lab 02 show the same pattern measured in a simulated mission.
03 · Technical Examples
Evidence you can inspect
The records behind each case, field by field. Numbered fields are the ones that matter; the notes say what each one tells us. All figures are simulated.
CostLab 01 · simulated figures
Two model requests that repeat the same work
Research’s summary call and Pricing’s extraction call, side by side, as Imperium recorded them.
- time
- 00:04.5
- agent
- research
- mission_id
- MSN-30101 (demo data)
- input.doc_hash
- sha256:9f2c…41ab
- usage.input_tokens
- 41,800
- usage.output_tokens
- 2,400
- cost.estimated
- $0.161
- result
- Tier B, 12% discount, net 45; 40 × XR-200 by 14 March
- time
- 00:11.4
- agent
- pricing
- mission_id
- MSN-30101 (demo data)
- input.doc_hash
- sha256:9f2c…41ab
- usage.input_tokens
- 41,800
- usage.output_tokens
- 2,100
- cost.estimated
- $0.157
- result
- Tier B, 12% discount, net 45; 40 × XR-200 by 14 March
= same document hash= 41,800 input tokens each+6.9 s later, after the summary was ready
Fields that matter
- input.doc_hash The same value in both requests: the model read the same two documents twice.
- usage.input_tokens 41,800 tokens each. The size is not the problem; Research’s call was the useful one.
- time and agent Pricing’s call started 3.3 s after Research’s summary reached the Coordinator, so the result already existed.
- cost.estimated $0.157 for facts the mission had. Observatory files it as finding F-L01-1.
Who supplied what
Imperium captured or governed both requests, attributed to agent, with usage, cost and the input’s content hash. Imperium does not yet record the mission id.
Observatory correlated and explained the matching hash and the timing into a duplicate-call finding, and the cost it added.
LatencyLab 02 · simulated figures
A trace with retries
Every inventory attempt Pricing made, with the re-plan call between attempts, on one timeline.
| Attempt | Start | Duration | Status | request.key | request.hash | Next step (model) |
|---|---|---|---|---|---|---|
| 1 | 00:03.6 | 6.0 s | 504 timeout | idem-7f31c2 | sha256:52be…0c19 | Retry the same request. |
| 2 | 00:11.7 | 6.0 s | 504 timeout | idem-7f31c2 | sha256:52be…0c19 | Retry the same request. |
| 3 | 00:20.2 | 6.0 s | 504 timeout | idem-7f31c2 | sha256:52be…0c19 | Retry the same request. |
| 4 | 00:28.9 | 6.0 s | 504 timeout | idem-7f31c2 | sha256:52be…0c19 | Retry the same request. |
| 5 | 00:37.6 | 2.4 s | 200 OK | idem-7f31c2 | sha256:52be…0c19 | Stock confirmed |
Fields that matter
- request.key and request.hash Unchanged across all five attempts: the same request, sent again. Nothing about the approach changed.
- Status and duration 4 timeouts of 6.0 s each. Infrastructure monitoring sees these as 504s with no caller.
- Next step (model) 4 re-plan calls, each answering “Retry the same request.” They add model cost to the wait.
- Critical path Validation, the draft and the approval could not start until stock was confirmed. This one tool held the mission.
Who supplied what
Imperium captured or governed each attempt with its agent, mission, attempt number, idempotency key and payload hash, and the re-plan calls between them.
Observatory correlated and explained the attempts into one retry loop, and the loop onto the mission’s critical path.
PolicyLab 03 · simulated figures
A DENY decision beside the tool response
The policy decision, the enforcement record and what the CRM returned, as three separate records.
- time
- 00:03.8
- agent
- crm-lookup (child of research)
- initiator
- svc-sales-automation · unverified
- policy
- restricted-customer-fields
- decision
- DENY
- fields.denied
- tax_id, credit_limit
- reason
- Restricted to finance roles
- time
- 00:04.0
- enforcement.status
- not confirmed
- enforcement.record
- none
- path
- connector → CRM directly (not via the enforcing gateway)
- time
- 00:05.2
- status
- 200 OK
- fields.returned
- customer_name, tier, contract_id
- fields.returned
- tax_id restricted
- fields.returned
- credit_limit restricted
- values
- not shown here
Fields that matter
- decision and fields.denied What should happen: the two restricted fields are not to be returned to this agent.
- enforcement.status What did happen at the boundary: nothing recorded. The connector bypassed the gateway that enforces the decision.
- fields.returned The overlap with fields.denied is the incident: the response carries both restricted fields.
- initiator The mission came from a shared service account. Who asked stays unverified, and the case says so.
In the main lab’s Sensitive data scenario the CRM’s field list is not recorded, so Observatory can say only that a denied request reached the CRM. In Lab 03 the response is captured at the connector, which makes the overlap visible.
Who supplied what
Imperium captured or governed the decision, the missing enforcement record and the response’s field list.
Observatory correlated and explained the three records into one finding, without treating the decision as proof that the request was stopped.
OutcomeLab 04 · simulated figures
Validation passed, approval rejected
What validation checked, what the inventory tool actually returned, and what the approver decided.
- time
- 00:08.8
- status
- passed
- checks.schema
- pass
- checks.totals
- pass
- checks.currency
- pass
- checks.terms
- pass
- checks.inventory_confirmation
- not checked
- time
- 00:03.6
- status
- 200 OK (from cache)
- result.stock
- 140
- result.source
- cached snapshot, 26 h old
- result.reservation
- none
- time
- 00:12.9
- decision
- rejected
- reason
- No stock confirmation for XR-200. Cannot send.
- business_outcome
- not achieved
Fields that matter
- checks.inventory_confirmation Not checked. Validation passed on what it looked at, and never looked at stock.
- result.source and result.reservation The stock figure was a day old and nothing was reserved: the requirement was never met.
- decision and reason The approver rejected the draft for exactly the requirement validation skipped.
- business_outcome Task completion, technical validation and the business outcome are three measurements. Only the last one failed.
Who supplied what
Imperium captured or governed the checks validation ran and skipped, the tool’s real answer and the approval decision.
Observatory correlated and explained a validation coverage gap and a business outcome not achieved, despite a completed mission.
From Insights
Further reading
Articles from Tracston Insights on the questions these investigations raise.
- GuideWho is watching the AI? Introducing AI Flow & Performance Monitoring (AFPM)The introduction: why AI work has to be reconstructed as one traceable flow across agents, models, policies, tools and infrastructure.
- InsightThe Agent Finished the Job. Nobody Knows What It Cost.A successful deployment hides a failed first attempt and a rebuild. Success and efficiency are different questions.
- InsightThe New Performance Metric: Cost per Successful OutcomeCounting compute is not enough. The number that matters is what it cost to get the right result.
- InsightWhen a MissionFlow Fails, Where Do You Start Looking?Every system can say it did its job. Finding the cause means seeing the whole story on one timeline.
- InsightYour AI Agents Are Talking to Each Other. Who Is Listening?Agents delegate, share context and act across systems. Each call can look fine while the chain is not.
- InsightYour AI Is Working. But Is the Process Working?A system can look healthy while its work is slow, wasteful or wrong. Judge AI by the whole process.
- InsightFrom Workflows to MissionFlowsA workflow fixes the path in advance. A MissionFlow fixes the goal and the limits, and the agent finds the path.
Read Insights: all AI Flow & Performance Monitoring (AFPM) articles AFPM definition in the Tracston Glossary →
How it works
Imperium captures. Observatory explains.
In Tracston’s AI Flow & Performance Monitoring (AFPM) solution, Imperium captures AI activity and its governance context. Observatory uses that telemetry to correlate events, investigate execution, and explain performance, cost, and outcomes.
Step 1 · Capture
Imperium captures and governs
Imperium records each AI request as it happens.
- Who started the mission, and whether that is proven
- Which agent acted, and on behalf of which parent
- Which tool or model it asked for
- Which policy applied, what it decided, and whether enforcement was confirmed
Step 2 · Explain
Observatory correlates and explains
Observatory receives that telemetry and correlates the events. Joining them into missions and steps is in development.
- Timing and dependencies along the critical path
- Retry and failure patterns
- Cost by agent, where the deployment exports it; by step in development
- Whether the result met the mission’s requirements (in development)
What this means in practice
The AI detail depends on Imperium
The AI investigation shown here relies on the telemetry Imperium supplies. Without it, Observatory still sees infrastructure signals, such as an HTTP 504, but not which agent, mission or policy they belong to.
Try the AI telemetry switch in the lab to compare.
Scoped to AI flows
This page covers the AI side of AFPM. Observatory’s broader infrastructure monitoring is a separate capability, described on the Observatory page.
Cost accountability
From AI spend to accountable work.
What did this mission cost?
MissionFlow records every model call a mission makes with its mission, step, tokens and cost. Imperium attributes each call it routes to a user and an agent, with its tokens and cost; attributing other agents’ calls to missions and steps is in development. Spend stops being one account total on an invoice.
What caused unnecessary spending?
Repeated searches, duplicate summaries and oversized context show up as patterns across calls. Each finding cites the events behind it, so the cause can be checked.
Did the output justify the cost?
Cost is read next to what the mission delivered and whether it was approved. Controls that cut waste can then be compared against the same result.
Real product screenshots
The products behind the simulation.
These captures of the actual Imperium and Observatory interfaces substantiate the capabilities above: attributing spend, finding its cause and judging the result. Unlike the lab, they show the real products, with demonstration data.
Investigate your own AI missions
See Imperium and Observatory working together on your agents, models and tools.