AFPM — AI Flow & Performance Monitoring
AI Optimization Should Be Continuous, Not a Quarterly Project
AI systems change every week, so keeping them efficient has to be a routine rather than a review every few months.
A building, once it is finished, mostly stays the way it was built. You inspect it now and then, fix what has worn out, and it carries on. A garden is different. Leave it alone for three months and it will not be the garden you planted. Things grow, spread and change on their own. A garden is looked after a little every week, or it is not really looked after at all.
Most organizations treat their technology like a building. Their AI behaves much more like a garden.
Why the old rhythm does not fit
Most organizations optimize systems after something becomes expensive or slow. The bill jumps, a customer complains, or a quarterly review flags a cost line that has grown. A team is assembled, the problem is fixed, and everyone returns to their normal work until the next surprise.
AI systems evolve too quickly for that model.
Prompts change, because people keep improving the instructions they give the AI. Models change, because providers release new versions and retire old ones, sometimes with little notice. Agents gain new tools, which means new ways to do things and new ways to take a long route. Users adopt new behaviors, asking for things nobody planned for. Workflows expand as teams add steps. And costs shift as prices, volumes and usage patterns move.
Each of these changes is small. Together, they mean that the AI system running today is not quite the one that was running last month.
There is a second reason the old rhythm fails. Each of these changes is usually approved on its own merits, by a different person, for a good reason. A product manager improves a prompt. An engineer adds a useful tool. A provider upgrades a model. Nobody is looking at the combined effect, because nobody owns it. So the drift is not the result of a bad decision. It is the result of many reasonable decisions that were never looked at together. A process that was efficient last month may become inefficient after a new model, policy or integration is introduced. An integration is a connection to another system, such as a new database or service the AI can now use.
This makes AI optimization a continuous operational discipline. Operational, because it belongs in the day-to-day running of the business, like watching cash flow, rather than in an occasional project.
A small change, a large bill
Imagine identifying that a particular workflow now uses 35% more tokens than it did two weeks ago. The infrastructure is unchanged. The output quality appears similar. Nobody has complained. But tracing the process reveals that a new agent instruction introduced an additional reasoning loop.
Or perhaps a powerful reasoning model is handling a classification task that a smaller model could perform reliably. Classification means sorting things into categories: is this email a complaint, a question or a sales lead? It is one of the simplest things AI does, and it rarely needs the largest, most expensive model.
These are small architectural decisions that can become very expensive at scale. “Architectural” here simply means a decision about how the system is put together, such as which model is used where and what instructions it gets.
What continuous optimization needs to see
Continuous AI performance management therefore requires several layers of visibility. Visibility simply means being able to see and measure something without special effort. Six layers matter most:
- Performance
- How fast the work is done, and whether it is getting slower.
- Quality
- Whether the results are good, as judged by people or by automated checks.
- Model selection
- Which model handles which step, and whether that choice still makes sense.
- Reasoning behavior
- How much thinking the AI does before it answers, and whether that is changing.
- Cost
- What each piece of work costs, including failed attempts.
- Business outcome
- Whether the work achieved what it was meant to achieve.
Looking at any one of these alone can mislead. Cost can fall because quality fell. Speed can improve because the AI stopped checking its work. Only side by side do they tell the truth.
This is also why the job cannot belong to one team. The finance team sees cost. The engineering team sees performance. The business team sees outcomes. Continuous optimization works when those views are brought together regularly, in the same conversation, with the same numbers in front of everyone.
It also requires a historical perspective. When a number moves, the first job is to explain why. That means being able to answer questions like these:
- What changed?
- When did it change?
- Which version caused it?
- Which model was active?
- Which policy was applied?
Each question depends on a record having been kept at the time. If nobody noted when the instruction changed or which model version was running, the investigation starts with guesswork.
What this means for your organization
Keep a change log for AI. Every time someone edits a prompt, switches a model, adds a tool or changes a policy, write down what changed, when and why. This single habit makes most later investigations quick.
Set a baseline for your top processes. For your three most important AI processes, record today’s cost per task, speed and quality. You cannot notice a change if you do not know where you started.
Review weekly, briefly. Fifteen minutes a week looking at a few numbers catches drift long before a quarterly review would.
Test changes before rolling them out. When someone wants to change an instruction or a model, try it on a sample of real work and compare cost and quality with the current version.
Ask the “smaller model” question regularly. For each step, ask whether a smaller, cheaper model could now do the job just as well. The answer changes as models improve.
- Measure.
- Compare.
- Understand.
- Improve.
- Repeat.
Tracston works on these questions in Observatory.
