[Research Briefing] AgentOps: A Framework for Controlling Uncertainty in Agentic AI Systems

Introduction

In enterprise environments, LLMs are evolving beyond simple chatbots into ‘agentic systems’ that autonomously plan and utilize tools. While these systems offer powerful capabilities to execute complex workflows autonomously, they simultaneously introduce significant operational risks due to unpredictability.

Traditional software engineering was dominated by deterministic thinking, where identical inputs yielded identical outputs. However, agentic AI is based on probabilistic reasoning and modifies execution paths at runtime based on generated code or the state of external tools. This causes new types of failures that cannot be captured through conventional log analysis or simple monitoring.

Specifically, in multi-agent environments where multiple agents collaborate, interaction complexity increases exponentially. The delegation of roles between agents, feedback loops, and changes in memory state render the system a black box, making it extremely difficult to identify the precise root cause during a failure.

Therefore, for enterprise-level AI adoption, it is essential to implement ‘AgentOps’—an operational framework that manages the entire system lifecycle beyond simple model tuning. This will serve as the core infrastructure to constrain the uncertainty of AI agents within a controllable range and reliably generate business value.

Core Arguments

This paper addresses the inherent uncertainty issues in agentic AI systems where complex LLM agents interact using memory, tools, and dynamic planning. It points out that traditional software observability practices struggle to resolve non-deterministic behaviors caused by probabilistic reasoning, fluid execution paths, and evolving memory states.

To solve this, the authors define the requirements for four key roles: developers, testers, site reliability engineers (SRE), and business users. Furthermore, they propose the ‘AgentOps Automation Pipeline,’ a six-stage process (Observe Behavior, Collect Metrics, Detect Issues, Identify Root Cause, Optimize Recommendations, Automate Runtime) that connects simple observation to automated optimization.

Analysis reveals that many organizations do not use dedicated observability platforms (8%), and 60% report that existing analysis tools do not meet their requirements. To bridge this gap, the paper suggests the need to extend standard protocols such as OpenTelemetry and explores the potential of graph-based analysis.

Ultimately, the goal is not to eliminate uncertainty entirely, but to ‘tame’ it. Specifically, the paper proposes designing a structure where the system can self-correct and improve in real-time by automating prompt calibration, workflow restructuring, and tool replacement.

Deep Dive into Core Failure Factors

First is the non-deterministic drift of execution paths. Agents establish plans dynamically by interpreting natural language instructions; due to the probabilistic nature of LLMs, they may select different sub-task decomposition methods for the same input. This causes intermittent failures that occur only in specific scenarios, making it impossible to verify all potential execution paths using traditional test coverage methods.

Second is the variability of memory and tool interactions. When an agent retrieves information from a vector database, subtle changes in context can lead to entirely different information retrieval due to the nature of similarity-based search. Additionally, if tools change or disappear at runtime, coordination failures occur where the agent fails to handle the change, resulting in infinite loops or incorrect tool calls.

Third is the collapse of feedback loops and guardrail violations. Agentic systems modify their behavior through internal reflection and user feedback. However, if the system ignores guardrail settings or fails to correct errors during reflection—thereby reinforcing incorrect reasoning—it optimizes behavior in unintended directions. This can lead directly to critical operational risks such as security policy violations or data leaks.

Source: Photo on Unsplash / Software Engineering Archive

Detailed Mechanisms of the 6-Stage AgentOps Automation Pipeline

The AgentOps automation pipeline consists of a six-stage process connecting raw behavior capture to self-healing mechanisms, forming a closed-loop structure to control system uncertainty. In the first stage, Observe Behavior, the system automatically instruments probabilistic actions such as LLM reasoning, tool usage, vector database queries, and user inputs, going beyond simple tracing. Specifically, it reconstructs dynamically generated decision paths by capturing execution flows driven by runtime-generated code, inter-agent communication, and feedback loops like internal reflection and guardrails. The resulting massive trace data is managed efficiently through statistical sampling or anomaly-based escalation techniques.

The second stage, Collect Metrics, transforms observed signals into structured insights. During the development phase, it tracks quality and flow metrics such as tool call frequency, memory access rates, task success rates, and reasoning depth. In the evaluation phase, it analyzes behavior distribution per execution round and deviation from golden traces; in the maintenance phase, it monitors regression and drift trends. From a business perspective, it measures system efficiency and reliability via cost, latency, ROI, and productivity metrics, providing customized insights by slicing data by time, task, or component.

In the third stage, Detect Issues, collected metrics are analyzed to identify both complete failures and subtle performance degradation. It detects task failures—such as incomplete outputs or responses that are syntactically valid but unusable—as well as component issues like LLM timeouts, low-confidence outputs, and inter-agent coordination failures. Specifically, it categorizes and assigns severity to adaptive loop collapses (e.g., guardrail violations or reflection feedback rejection), metric-based failures due to threshold breaches or pattern shifts, and security/compliance issues like access control policy violations, triggering smart alarms.

The fourth stage, Identify Root Cause, bridges the gap between symptoms and solutions. Analysis targets are divided into LLM-related issues (ambiguous prompts, hallucinations), interaction protocol problems (incorrect tool selection, syntax errors), and flow/coordination failures caused by task decomposition inconsistencies or structural defects. External factors such as input injection, goal drift, and embedded bias are also analyzed. To achieve this, the system utilizes comparative views of normal vs. failed traces, causal path explorers, and analysis-dedicated chat interfaces to answer “Why did it fail?”

In the fifth stage, Optimize Recommendations, specific improvement plans are generated based on the analyzed causes. This includes prompt tuning to remove ambiguity and apply structural patterns, workflow optimization such as improving task decomposition and parallelization, and call optimization to remove unnecessary calls and apply smart retry logic. It also enhances resilience by introducing fallback options and strengthening guardrails, recommending the optimal combination considering SLA trade-offs between quality, performance, and cost.

The final stage, Automate Operations, completes the closed-loop by applying high-confidence improvements directly to the system. It adjusts behavior by augmenting prompts, runtime data, and tool instructions in real-time or tunes configuration thresholds and retrieval logic. If critical issues persist, it automatically performs LLM model replacement, tool replacement, workflow modification, or guardrail updates without requiring code changes or redeployment. For example, if an SRE agent is observed using diagnostic tools inconsistently, the system detects the issue, identifies inaccurate prompt instructions as the cause, automatically applies an optimized prompt, and continuously monitors the result, realizing a self-correcting structure.

Evolution of the 4 Key Roles and Responsibilities in Agentic Systems

The introduction of agentic systems blurs the boundaries of traditional software development and operations, fundamentally expanding the scope of responsibility for four key roles. First, the Developer must move beyond breakpoint debugging based on fixed execution paths. Because agentic systems are non-deterministic—producing different results from the same initial conditions due to probabilistic reasoning—developers must focus on precisely tuning LLM parameters (e.g., temperature, context window) and building instrumentation to capture structured trace data. It is essential to conduct iterative experiments within the vast unstructured design space created by prompt engineering and maintain adaptability to runtime-generated code and execution flows.

The Tester’s role shifts from binary verification of code coverage or requirement fulfillment to the analysis of a ‘performance spectrum.’ In agentic systems, executing all code paths does not guarantee the verification of all possible behaviors, and success criteria appear as a continuous range depending on the application’s criticality. Consequently, testers must meticulously verify intermediate states and decision points, such as tool selection and routing logic, rather than just final outputs. Additionally, they assume the responsibility of maintaining a post-deployment monitoring-based testing framework to ensure reliability and performance as the system evolves through use cases and learning.

The Site Reliability Engineer (SRE) evolves from simple incident response and performance monitoring to an ‘active trend analyst’ who diagnoses system health in real-time. SREs must track not only numerical metrics—latency, cost, tool usage, human input—but also semantic indicators to capture early signs of systemic issues. This approach is akin to preventing a heart attack via vital signs rather than diagnosing it after it occurs. Upon detecting drift or anomalies, they establish mitigation strategies such as dynamic reconfiguration, prompt changes, or tool replacement, completing the closed-loop between observability and continuous system improvement.

Finally, the Business User evaluates system value through business-centric metrics—revenue, cost, ROI, customer satisfaction—rather than technical details. They focus on optimizing the trade-off between cost, latency, and quality. Business users expand their role to analyzing the root causes of detected issues in relation to business outcomes and supporting strategic decisions through ‘What-if’ analysis or A/B testing.

All four roles must shift from static code management to guiding and improving an entity that possesses state and evolves. Developers pursue design flexibility, testers diversify verification, SREs ensure proactive responsiveness, and business users focus on value-driven optimization. These changes are integrated through the new discipline of AgentOps, where data and feedback from each role lead to system self-correction via the automation pipeline.

Consequently, operations in agentic systems require a completely different approach than traditional ITOps. As the system adjusts its own prompts or configurations, verifying and controlling these changes becomes a common challenge for all roles. Automation serves as the bridge allowing users without deep ML expertise to perform these complex operational processes efficiently, ultimately constraining uncertainty within a controllable range for safe enterprise operation.

Agent-Specific Failure Taxonomy and Root Cause Analysis (RCA)

Issue detection in agentic systems requires a precise process of defining failure types and scope, assigning severity, and performing correlation analysis of related events. This paper categorizes failures into three main buckets. First are Task failures, which include cases where a task was not completed, outputs were inaccurate or only partially generated, or responses were syntactically valid but practically unusable.

Second are Component issues. While these may not completely block task completion, they serve as indicators of serious systemic problems. Specifically, these include LLM timeouts, low-confidence outputs, tool and vector database errors, and inter-agent coordination failures. These issues act as signals for potential system performance degradation.

Third are Failures in handling feedback. These are classified as critical errors that destroy the agent’s intelligent adaptation loop. This includes guardrail violations, rejection of feedback during internal reflection, decreased user satisfaction, or collaboration failures. It signifies that the agent has lost the ability to self-correct and respond to external inputs.

Additionally, metric-based failures exist. Failures are identified when monitoring values exceed defined thresholds, exhibit anomalous values, or show change points indicating discontinuous shifts in behavior. Security and compliance issues, such as access control policy violations, sensitive data exposure, resource misuse, and SLA breaches, are also essential failure categories to manage.

Root Cause Analysis (RCA) for these symptoms is divided into four categories. The most frequent are LLM-related issues, including instruction violations, ambiguous or inaccurate prompts, intent recognition errors, and hallucinations. Even a minute defect in a prompt can lead to significant differences in agent behavior.

Next are Interaction protocol errors. These occur during inter-agent communication or tool calls, caused by inappropriate tool/parameter selection, syntax errors, or incorrect configurations. This often leads to execution flow interruptions when required context is missing or retrieval fails.

Third are Flow and coordination failures. These are deeper behavioral issues caused by inconsistencies in task decomposition, inconsistent coordination between agents, structural gaps, or overly complex multi-stage goal settings. These occur when the system design conflicts with the agent’s autonomous reasoning path.

Finally, there are failures due to External factors. These include deviations in planning caused by input injection, goal drift where system behavior departs from intent, embedded bias that distorts decision-making, and policy overrides that suppress necessary verification processes. AgentOps supports the identification of these causes through comparative views of normal vs. failed traces, causal path explorers, and analysis-specific chat interfaces.

Technical Directions and Standardization Strategies for Uncertainty Control

To effectively control the uncertainty of agentic systems, a standardization strategy is essential to move beyond fragmented observability tools. Currently, the agent analysis environment is split between developer-centric GenAI tools and SRE-centric observability platforms, resulting in a lack of interoperability. To resolve this, the paper suggests extending OpenTelemetry (OTel)—the standard for logs, traces, and metrics—to agent-based workflows. Efforts like Traceloop’s OpenLLMetry are driving this standardization by supporting frameworks such as LangGraph, CrewAI, and AutoGen.

Furthermore, the paper emphasizes adopting protocols that unify interaction and tool usage patterns among agents beyond simple data collection. The Model Context Protocol (MCP) and Unified Intent Mediator (UIM) protocol serve as key foundations for enhancing interoperability between different agentic systems. This standardization provides a consistent data framework across the entire AgentOps process—from instrumentation to evaluation and automation—creating an environment where enterprises can optimize systems without vendor or framework lock-in.

From a technical analysis perspective, the unique nature of data generated by agentic systems must be considered. Agent behavior is not a simple linear flow but a graph-structured entity where dynamic planning and tool calls are intertwined. Therefore, graph-based analysis techniques capable of encoding semantically rich alphanumeric data are required, rather than traditional text-based log analysis. Approaches such as Graph Neural Networks (GNN) offer a promising direction for detecting issues and analyzing root causes by reflecting the structural characteristics of agents.

The final objective is to build a self-healing and adaptive execution system where the system resolves problems without human intervention. This means more than just fixing errors; it refers to a mechanism that finds the optimal path in real-time at runtime. For example, if tool call failure or performance degradation is detected, the system automatically reroutes the task, adjusts LLM parameters, or immediately changes the execution plan to minimize the impact of suboptimal behavior.

This self-healing mechanism is completed when combined with ‘Automated Operations,’ the final stage of the AgentOps pipeline. The system automatically performs actions such as calibrating prompts, updating guardrails, or replacing the LLM model itself based on observed behavior data. This enables a structure where the system improves performance in real-time without code changes or redeployment, implementing a strategy to efficiently ‘tame’ rather than completely eliminate the intrinsic uncertainty of intelligent systems.

Ultimately, future agentic AI operations will evolve around three pillars: data integration through standard protocols, deep graph-based analysis, and self-healing automation. These technical directions ensure that agents operate safely and adaptively in complex enterprise environments, securing system reliability and efficiency while minimizing operator intervention.

Operational Checklist for AgentOps Framework Adoption

To proactively prevent system collapse in practical engineering pipelines, the following core defense mechanisms must be enforced at the operational stage:

• Dynamic Tracing Implementation: Are all paths, including LLM reasoning, tool calls, and vector DB queries, being captured?
• Role-Based Metric Definition: Are customized metrics set for developers (success rate), SRE (latency/cost), and business users (ROI)?
• Non-Deterministic Test Design: Is there a testing framework that verifies intermediate decision points and tool selection logic rather than simple output comparison?
• Failure Taxonomy Establishment: Have agent-specific failure types, such as prompt errors, tool misuse, and coordination failures, been defined?
• Causal Analysis Tools: Are there tools capable of tracing the causal path from symptom to root cause?
• Optimization Recommendation Mechanism: Does the system provide specific solutions, such as prompt revisions or workflow changes, for detected issues?
• Runtime Auto-Correction: Is there a process to automatically update prompts or configurations without code deployment when confidence is high?
• Standard Protocol Compliance: Have standard semantic conventions for interoperability, such as OpenTelemetry, been applied?

Related Work

The current agent observability and analysis ecosystem is evolving into two main categories. First are GenAI-specific tools such as Phoenix, LangFuse, and LangSmith, designed primarily for AI developers. These are specialized for tracing and debugging LLM-based applications and are optimized for fine-tuning and experimenting with model responses during development.

Second are traditional observability platforms such as Datadog and IBM Instana. Originally designed for general software infrastructure visibility, they have evolved by expanding features to allow SREs to operate agentic systems reliably. These platforms excel at integrated monitoring of overall system performance metrics and infrastructure health.

Despite these tools, significant limitations remain in handling the unique characteristics of agentic systems. Regarding data collection, efforts continue to extend standard protocols like OpenTelemetry, with Traceloop’s OpenLLMetry supporting the observability of frameworks like LangGraph, CrewAI, and AutoGen. Efforts from OpenInference and LangFuse also persist.

Nevertheless, standardized semantic conventions for instrumenting high-level behaviors such as ‘planning’ or ‘reflection’—the core of agentic systems—have not yet been widely adopted. The lack of a standard framework for semantic analysis of why an agent made a specific decision, beyond simply leaving logs or traces, remains a major constraint.

Severe gaps are also found in the analysis stage. Current tools lack standardized failure taxonomies. Specifically, complex failures like ‘unintended infinite loops’ in multi-agent workflows are extremely difficult to detect with existing tools. While research into encoding structural agent data using Graph Neural Networks (GNN) is underway, practical application to failure analysis is still in the early stages.

Root Cause Analysis (RCA) capabilities are also insufficient. There are few tools that effectively capture the causal relationships between an agent’s decision, a tool’s action, and the finally observed failure. While causal inference in process mining and causal discovery in agentic systems provide a foundation, practical implementation in operational tools is rare.

Finally, there is a notable absence of recommendation systems that provide actionable solutions beyond simple issue detection. Current tools are limited to notifying users that latency has increased or hallucinations have occurred. They rarely provide guidance on what specifically to modify, such as prompt tuning, agent structural redesign, or parameter changes.

Consequently, users must manually adjust trade-offs between cost, latency, and quality, and humans must intervene for every recurring issue. While initial research into automation approaches specialized for agentic systems has begun, most operational workflows still rely on manual intervention and have not reached a level of automation where the system refines itself precisely.

Failure Taxonomy

To increase the operational efficiency of agentic systems, it is essential to clearly define and categorize failures. However, current agent analysis tools lack standardized failure taxonomies that systematically categorize failure types beyond basic observability. This means the way failures are defined and shared varies by organization or tool, posing a significant obstacle to establishing consistent response strategies.

While early-stage classification proposals exist in academia and certain research—such as the prior studies the paper cites—these have not yet been widely adopted or established as industry standards. Without a standardized system, many operators rely on intuition to handle failures, leading to a sharp increase in management costs as system scale grows.

Specifically, the absence of a standardized taxonomy severely impacts the detection of complex failures in multi-agent workflows. For example, ‘ping-pong’ requests between agents leading to infinite loops are critical issues unique to agentic systems, yet they often go unnoticed because there are no standardized criteria to define and detect them systematically.

The lack of taxonomy is more than a nomenclature problem; it directly leads to functional limitations in analysis tools. Without standardized failure types, it is difficult to design algorithms that automatically classify or group specific failure patterns from collected data. Analysts must manually review numerous trace data points, which is a primary cause of increased Mean Time to Recovery (MTTR).

Furthermore, this gap extends to interoperability issues between tools. When using observability tools from different vendors, the definition of ‘failure’ in one tool may be interpreted differently in another. This causes communication errors between SREs, developers, and business users, making it impossible to establish a common metric for assessing system stability.

Consequently, realizing AgentOps requires the establishment of a universal failure classification standard that reflects the behavioral characteristics of agentic systems, moving beyond simple data collection. Only then can a full automation pipeline be implemented—one that automatically assigns severity during issue detection and maps precise solutions during RCA.

Current analysis tools remain focused on traditional logs and metrics, failing to systematically capture ‘semantic failures’ such as errors in an agent’s reasoning process or tool selection. The original text emphasizes the need for this standardization and suggests that standard protocols like OpenTelemetry should expand to support agent-based workflows.

Root Cause Analysis and Recommendation Systems

To enhance the operational efficiency of agentic AI systems, precise root cause analysis and the provision of solutions are essential beyond mere issue detection. However, looking at the current agent analysis landscape, Root Cause Analysis (RCA) technology remains insufficient. Existing tools fail to effectively capture the complex causal relationships between an agent’s decision, the actual behavior of external tools, and the final observed failure.

Especially in multi-agent systems, the primary cause of failure is often a causal chain reaction occurring during agent interaction rather than a single-point error. However, due to a lack of standardized methodologies for analyzing this, operators only see the symptoms and struggle to understand the deep causal structure of ‘why’ a certain result was produced.

This lack of analytical capability naturally leads to a void in Recommendation Systems. Many analysis tools currently on the market are limited to flagging superficial problems such as increased latency or occurrences of hallucinations. The problem is that a massive technical gap exists between detecting an issue and suggesting specific measures to resolve it.

What operational practitioners truly need are not simple alerts, but actionable suggestions that can be applied immediately. For example, when repetitive failures occur in a specific task, the system should provide concrete guidelines on how to tune the prompt, how to restructure the agent, or which LLM parameters to change.

As pointed out in the original text, examples of providing this level of optimization suggestions are very rare. Most systems rely entirely on manual intervention by human experts for the steps following the capture of problem signs—namely, deriving and applying the solution. This becomes a factor that exponentially increases operational costs as the scale and complexity of agentic systems grow.

Ultimately, for the data provided by observability tools to lead to actual system improvement, the pipeline of ‘Detection $\to$ Root Cause Analysis $\to$ Optimization Recommendation’ must be organically connected. To realize the true value of AgentOps, a system must be established that goes beyond telling us “what went wrong” and provides automated answers to “how to fix it.”

In conclusion, while existing GenAI analysis tools have focused on developer-centric tracing and simple monitoring, future AgentOps must evolve to strengthen RCA capabilities that clarify the causal relationship between agent decision paths and tool behavior, and build intelligent recommendation systems that suggest specific tuning plans based on this analysis.

Author’s Perspective and Practical Advice

This paper presents the perspective that AI agents should be treated not as simple ‘software modules’ but as ‘employees to be managed.’ In practice, when building multi-agent systems, the most painful point is the difficulty in finding the answer to ‘why is it suddenly behaving like this,’ and it is impressive that this has been formalized into an operational framework called AgentOps.

Particularly striking is the analogy of the SRE as a ‘doctor.’ The logic that one should respond proactively by monitoring vital signs—referred to here as semantic indicators—rather than responding after the system has completely collapsed is, in my view, the gold standard for enterprise AI operations.

However, the ‘automation’ stage of the proposed six-stage pipeline could be very risky. Discussions on specific safety measures or rollback strategies for unexpected side-effects that may occur in a loop where AI modifies AI should be further supplemented.

Ultimately, the core of AgentOps is ‘visibility.’ The success of the commercialization of agentic AI will be determined by how much the interior of the black box can be visualized and analyzed in a structured graph form.

Conclusion and Enterprise Governance Guide

To ensure that internal AI projects do not remain as early-stage PoC demos but settle into actual business value, the following governance principles must be established from the planning stage:

1. Visibility-First Principle: All agent decision-making processes and tool usage histories must be recorded in a traceable and analyzable form.
2. Role-Based Monitoring Principle: System performance metrics must be separated into technical metrics and business value metrics, provided to their respective owners.
3. Incremental Automation Principle: Auto-optimization and correction features must start with ‘Human-in-the-loop’ requiring human approval and transition to full automation only when confidence is verified.
4. Standardization Compliance Principle: Adopt open observability standards such as OpenTelemetry to ensure interoperability between systems and avoid vendor lock-in.
5. Continuous Verification Principle: Define post-deployment monitoring not as simple failure detection, but as a continuous testing process that tracks the behavioral changes of evolving agents.

References and Original Analysis Materials