Closing the Last Mile of AI Observability in Regulated Industries

Regulators work from evidence

In regulated industries, AI agents raise the stakes for observability, because every step they take should make sense to the engineers who run the system and hold up for the examiners who review it. When an insurer or bank is examined, examiners want to know who accessed a record, when it happened, what changed, and whether the right controls were enforced. With AI in the loop, they also ask what the agent did, which data it touched, and whether it was allowed to.

Agents also make those questions harder to answer. A single agent decision can involve several model calls, tool calls, and systems of record, and the same input does not always produce the same steps. The record of that activity must be complete, so no step is missing. It needs to be tamper-evident, so an examiner can trust it hasn’t been changed. And it has to protect the sensitive data it contains, so the audit trail doesn’t become a privacy risk of its own. Standard observability tools weren’t designed to meet all three requirements. Security tools such as a SIEM or a locked log archive help with storage and alerting, but they can only keep what applications send them. On their own, they don’t capture every agent step with the context an examiner needs.

This is one reason enterprise AI stalls in the last mile, the gap between a general-purpose model and the data, rules, and workflows of a real business. Foundation models arrive without a business’s context, its systems of record, or any way to trace their decisions back to the policies they were meant to follow. A pilot can perform well and still stall before production if no one can show what the agent did on each run.

The AIVista Platform helps close this gap by producing that evidence as agents work. AIVista builds the platform, and NTT DATA engineers, including forward-deployed engineers (FDEs) who work directly with each client, deploy and configure it in the client’s environment. On the platform, a workflow is a business process, such as reviewing an insurance claim or processing a policy change, that the platform runs as a series of steps. Each step can call an API, run an agent, or have an agent call a tool, such as a client’s claims or policy system. Because any of those steps can touch regulated data, the platform records them as they happen, with observability and audit built into every layer. 

This post explains how the platform does that, starting with a distinction many platforms blur, the difference between telemetry and audit.

Two kinds of evidence, often confused

Modern distributed systems produce evidence in two ways, through telemetry and through audit. The two are often treated as interchangeable, but they are not. Keeping them as separate systems is one of the most important design decisions an AI platform will make. Let’s understand that distinction, starting with telemetry.

Telemetry: seeing how your system behaves

Telemetry is the continuous stream of signals a system emits about its own behavior. It takes three forms.

• Logs are timestamped records of discrete events, such as an error, a login, or a failed connection.
• Metrics are numeric measurements over time, such as latency, error rates, throughput, and resource usage.
• Traces follow a single request across services, showing every hop, every dependency, and where the time went.

Together, these answer the question “What happened?” When something breaks at 2 a.m., telemetry tells you where, why, and how far the damage spread.

For an AI platform, telemetry also needs to track AI-specific activity. On the AIVista Platform, that activity carries two identifiers. A trace identifier (trace ID), assigned automatically by OpenTelemetry, an open-source standard for collecting telemetry, follows a single request through every service it reaches. A workflow run identifier (workflow run ID) covers one run of a workflow from start to finish, which can take hours, and groups the many traces inside it, such as fetching data, processing a document, and storing the result. The two answer different questions. “Why did this call fail?” needs the trace ID, while “What did this agent do on this case?” needs the workflow run ID. Both travel from the API through the workflow interpreter (the component that runs each workflow step by step), agents, model calls, and tool calls. The audit service records them too, so an examiner’s question can later be tied back to the technical record.

All of this telemetry is built on OpenTelemetry and sent to the native monitoring tools of whichever cloud the platform runs on, so no service code is tied to a specific cloud. However, telemetry has one property that matters for compliance. It is built for operations, not evidence. It is often sampled to control cost, aggregated to stay manageable, and kept for days or weeks rather than years. It is designed to help teams debug and tune systems, not to serve as a long-term record for examiners.

Where telemetry falls short

Imagine a state examiner following up on a consumer complaint with this request.

“Show me everything the AI agent did on this claim over the last five years, and prove none of it has been altered.”

Telemetry can’t reliably answer that request, for four reasons.

• Telemetry is often sampled. To keep costs down, many systems record only a fraction of requests, so some accesses are never captured. 
• Retention is short. Logs and traces are usually kept for days or weeks, so a five-year history is long gone. 
• Telemetry usually sits in mutable storage. Logs are written to ordinary log services or databases, where anyone with the right administrative access can edit or delete entries, and nothing in the log shows that a change was made. That makes it impossible to prove that a record is the original.
• Telemetry lacks the context an examiner needs. A log line can show that a request succeeded (for example, that it returned HTTP 200), but not who made it, under what authority, or which customer’s data it touched. Logs are also written by many teams in different formats, so even the context that does exist is hard to piece together.

You could turn off sampling, keep logs longer, and lock the storage. At the volume telemetry produces, that gets expensive, and it still doesn’t add the missing context. The record examiners rely on needs its own system.

Additionally, frameworks for regulated industries such as NYDFS Part 500, the NAIC Model Bulletin, and the EU AI Act need more from record-keeping. For instance, NYDFS Part 500 (section 500.6) requires systems that can reconstruct material financial transactions, with those records kept for at least five years. The NAIC Model Bulletin on insurers’ use of AI expects insurers to document their AI systems and controls and produce that documentation during an examination. The EU AI Act requires high-risk AI systems to log events automatically and providers and deployers to keep those logs for at least six months, with obligations for most stand-alone high-risk systems now applying from December 2, 2027. Meeting those requirements takes a dedicated audit service.

The audit service: proving the right controls were enforced

An audit service has one job, which is to capture and preserve a trustworthy record of security- and compliance-relevant events. Telemetry answers “What happened?” The audit service answers “Who did what, to which data, and when?”

At AIVista, we hold ourselves to a simple rule. Telemetry can never stand in for an audit event. Every audit event follows its own path through the platform, separate from telemetry. That path has three goals. It must capture every relevant action without gaps, store each record so it can’t be changed or deleted without detection, and make the record searchable and verifiable without exposing the regulated data inside it. The seven steps below show how the platform meets each goal.

Inside the AIVista audit service 

1. Capture. When a platform component performs a governed action (such as an API call, a workflow step, or an agent or tool call), it triggers a structured audit event through our audit SDK. The event records the actor (a person or service), the agent and who delegated to it, the tenant, the workflow, and the action, plus the same trace ID and workflow run ID used by telemetry. Unlike telemetry, audit events are never sampled, so the record of governed actions is complete.
2. Accept or fail loudly. An audit gateway authenticates the caller and validates the event. It then waits for a durable message queue (Kafka) to confirm the write before reporting success. If the queue can’t confirm, the gateway fails closed. The action is reported as an error, and the event is never quietly marked as accepted. That way, no action can succeed without leaving a record. The trade-off is availability. If the queue is down, the action fails instead of running without a record.
3. Encrypt per record. A consumer service encrypts each event’s payload with its own AES-256-GCM key, the encryption standard banks and cloud providers use to protect sensitive data. That key is then wrapped under two separate recovery keys held in the cloud’s key management service. Audit records often reference regulated data, so they have to be protected. A separate key for every record also makes it possible to erase a single record for GDPR, as described below.
4. Write to three stores. Each event is written to three places, because no single store can do all three jobs, and the proof of a record must sit apart from the record itself. If one system held both, anyone who could change the record could also change the proof. Each store has a different job.

• an immutable object archive (write-once, object-lock storage) that holds the encrypted record under retention locks that block edits and deletes during the retention period, though it can’t be searched;
• an append-only Merkle-tree log, a running fingerprint of every record, so any later change shows up. It is built on Trillian, an open-source project from Google. It holds only a hash of each record, not its content, so the log itself never exposes regulated data;
• a PostgreSQL index that holds each record’s labels, never its content, and makes the trail searchable, because an examiner’s request is a query and neither of the other stores can answer one.

The writes happen in order, and the pipeline marks an event as processed only after all three succeed, so the searchable index never points to a record that isn’t already stored and hashed. If a write fails, it is retried safely, and events that keep failing go to a dead-letter queue where operators can see them.

1. Anchor the log. A dedicated signer service sequences each new record hash into the Merkle log and continuously copies the latest tree head (a signed summary of the whole log) to write-once storage. From then on, any change to an earlier record can be detected. Because the tree heads sit in separate write-once storage, even someone with access to the log can’t rewrite its history without the change showing up.
2. Verify without exposing. When an auditor or examiner needs a workflow’s trail, a reader service rebuilds it, recomputes each record’s hash, and checks the Merkle inclusion proof. It reports whether the record is intact without decrypting the payload, so integrity can be proven without regulated data being shown.
3. Separate the duties. Sensitive administrative actions require approval from at least two other administrators through a dedicated admin service. These include retiring a certificate the platform uses to sign events, changing the master key that protects every record key, and removing an auditor’s access to records. Retention can’t be changed at all. Each record’s disposal date is set when the record is written, based on the retention period configured at deployment. As a result, the people who run the system can’t quietly rewrite the record of what they did.

Together, these steps give examiners the records they request and give auditors proof that the controls behind those records held. The resulting evidence is complete, immutable, kept as long as regulation requires, access-controlled, and encrypted. Because each record is encrypted with its own key, an individual record’s payload can be made permanently unreadable by destroying that key. Where no legal duty to keep the record applies, this lets a client honor a GDPR erasure request while the Merkle log stays intact and still proves that no other record was changed.

Two tools, one goal

It’s tempting to make one system do both jobs. In practice, using telemetry as your audit trail leaves gaps, and using an audit service as a debugging tool adds cost and complexity. The two are optimized for different goals.

The real value comes from connecting them. Every AIVista audit event carries the same trace ID as the telemetry for that request. The trace ID is part of each audit record, and that record’s hash is anchored in the signed Merkle log, so the link between an audit record and its telemetry is tamper-evident too. While the telemetry is still retained, an examiner’s or auditor’s question can be tied straight to the full technical history of the request behind it. After that, the audit record has to stand on its own, which is why it captures the actor, action, and workflow. An engineer looking into an anomaly can move from a suspicious trace to the audit record for that request, which shows who acted, on what, and when. As a result, operations and compliance teams work from connected records rather than separate silos. 

One design, on all three major clouds

Regulated enterprises rarely run on a single cloud, and data-residency rules often decide where a workload can run. So we built AIVista’s observability and audit layers to be provider-neutral by design.

• Shared application code doesn’t hardcode a cloud provider. Cloud-specific behavior lives in adapters behind common interfaces, and configuration picks the adapter at startup.
• Telemetry attributes and the audit event schema are identical on every cloud.
• Every cloud gets the same audit guarantees, using its own native building blocks.

We built the full observability and audit stack on Azure first. We then brought the same design to AWS and Google Cloud. On both, live end-to-end tests against real workflow runs confirm that audit trails can be retrieved and their integrity verified. We don’t call a step done until it’s deployed, verified against the current code, and backed by test evidence.

Making compliance a continuous property

Compliance has traditionally been a periodic scramble. An exam or audit is announced, teams spend weeks pulling logs and building spreadsheets, and everyone hopes nothing was missed.

Platforms that invest in both telemetry and a dedicated audit service change that. Evidence is produced continuously, as a natural result of how the system runs. On the AIVista Platform, producing audit-ready evidence is part of everyday operation rather than a separate project. That evidence covers logging and record-keeping, one part of what examiners review alongside areas such as model risk management.

That is how the AIVista Platform helps regulated businesses move AI from pilot to production by producing the evidence examiners ask for as the work happens.

Building AI for a regulated industry? Contact the NTT DATA AIVista team to see how the AIVista Platform can produce audit-ready evidence for your AI workflows from day one.