The Data Pipeline, Not the Model, Emerges as the Bottleneck in AI-Assisted Root Cause Analysis

A burgeoning consensus among observability engineers indicates that the core reasoning capabilities of large language models (LLMs) are no longer the primary impediment to effective AI-assisted root cause analysis (RCA). Instead, the more formidable challenge has shifted to the intricate data pipeline responsible for curating and delivering relevant information to these advanced models. This paradigm shift suggests that strategic investments in context preparation may yield greater returns than simply deploying larger or more powerful LLMs.

The Evolution of Incident Response and the Rise of AIOps

For decades, incident response has been a critical yet often reactive discipline within IT operations. As systems grew more complex, distributed, and dynamic, the volume and velocity of operational data exploded. Traditional methods of manual log analysis, dashboard monitoring, and alert correlation became increasingly insufficient, leading to prolonged mean time to resolution (MTTR) and significant business impact. This environment gave rise to the field of Observability, which aims to provide deep insights into the internal states of systems by collecting and analyzing telemetry data – logs, metrics, and traces – allowing engineers to understand "why" something is happening, not just "what."

The advent of Artificial Intelligence for IT Operations (AIOps) marked a significant leap forward, promising to automate and enhance various aspects of incident management, including anomaly detection, predictive analytics, and, crucially, root cause analysis. Early AIOps solutions often relied on machine learning algorithms to identify patterns and correlations in data. However, the emergence of sophisticated LLMs, with their unparalleled ability to understand natural language, process vast amounts of text, and perform complex reasoning tasks, introduced a new frontier for automating RCA. These models held the promise of not just identifying symptoms but articulating causal chains and suggesting remediation steps, much like a human expert.

The Shifting Bottleneck: From Reasoning to Context

Initially, the focus in AI-assisted RCA was heavily placed on the LLM itself – its size, its training data, its architectural sophistication, and its inherent reasoning prowess. The assumption was that a more capable model would inherently lead to better RCA outcomes. However, practical deployments have revealed a different reality. Engineers have observed that even the most advanced LLMs struggle when fed incomplete, noisy, irrelevant, or poorly structured data. This has led to the realization that an LLM’s output quality is intrinsically tied to the quality and relevance of its input, echoing the long-standing computing adage, "garbage in, garbage out."

This realization marks a critical pivot in the industry’s approach to AI-driven RCA. The "harder problem" is now recognized as residing within the "harness" – the sophisticated data pipeline and context engineering mechanisms that determine what information reaches the model and in what optimal format. This involves pre-processing, filtering, correlating, and structuring diverse telemetry streams into a concise, high-signal context that an LLM can effectively interpret and act upon.

Two Dominant Approaches to AI-Assisted RCA

Currently, most AI RCA initiatives broadly fall into two methodological categories, each with distinct advantages and challenges:

1. Agent-based Designs: In this approach, the LLM acts as an autonomous agent, equipped with a suite of tools and an iterative reasoning loop. The model is given an initial prompt or anomaly and then "investigates" by choosing what telemetry data to fetch, what commands to execute (e.g., query a log database, check a metric dashboard, ping a service), and how to interpret the results. It dynamically forms hypotheses, tests them, and refines its understanding through successive interactions with the operational environment.

   • Advantages: This method offers significant flexibility. An agentic LLM can potentially uncover novel issues or correlations not anticipated by predefined rules or correlation sets. It can adapt its investigative path based on emergent findings, mimicking a human engineer’s exploratory diagnostic process.
   • Disadvantages: The primary drawback of agent-based systems lies in their operability and debuggability. Accounts from companies like ZenML and Incident.io, who have explored multi-agent LLM investigations, describe them as notoriously difficult to troubleshoot in production. A failed diagnosis doesn’t leave a clean stack trace; instead, it often results from unpredictable prompt interactions, emergent coordination failures between agents, or the LLM making an incorrect decision about what data to fetch next. This inherent non-determinism makes it challenging to pinpoint whether a diagnostic failure stemmed from the model’s reasoning or from its flawed data acquisition strategy.

2. Deterministic Designs: This camp prioritizes pre-correlation and structured context delivery. Before engaging the LLM, these systems meticulously correlate diverse signals (logs, metrics, traces, configuration changes, topology maps) upfront. They then present the LLM with a single, highly focused, and prepared context, often representing a potential causal chain or a set of strong indicators. The LLM’s role is primarily to reason over this curated dataset to pinpoint the root cause, articulate the cause-and-effect chain, and suggest fixes.

   • Advantages: Deterministic designs offer superior repeatability and cleaner evaluation. When a diagnosis fails, engineers can more confidently attribute the error to the model’s reasoning capabilities, as the input data is known and controlled. This makes debugging and improving the system much more straightforward. The pre-correlation work also often leads to lower token costs, as the LLM receives a compact, high-signal prompt rather than needing to process raw, voluminous data or generate numerous intermediate queries. Companies like Dynatrace, with its Davis AI, exemplify this approach, leveraging deterministic, topology-based causal analysis to traverse real-time dependency maps and pinpoint root causes.
   • Disadvantages: The main trade-off is reduced flexibility. If the initial correlation pipeline misses a critical signal or fails to identify a novel correlation, the LLM will never see that crucial piece of evidence, potentially leading to an incorrect diagnosis. This approach is most effective when incidents fall within a somewhat predictable pattern or when the underlying topology and dependencies are well-mapped.

Coroot’s Research: Isolating the Variables

Recognizing the ambiguity in attributing diagnostic failures in LLM-powered RCA, observability vendor Coroot undertook original research to disentangle the performance of the LLM’s reasoning from the efficacy of its data harness. Engineer Nikolay Sivko’s work aimed to answer a more precise question than "Can AI do RCA?" by separating the two distinct jobs: the LLM’s reasoning over provided data and the harness’s role in data selection and formatting.

Coroot’s methodology employed a deterministic pipeline. It correlated various signals into actionable "findings" and presented them to the LLM in one focused context, crucially without an agent loop. This design ensured that if the LLM produced a wrong answer, the blame could be squarely placed on the model’s reasoning, not on missing or incorrectly fetched evidence.

To test this, Sivko constructed a specific incident scenario: a Chaos Mesh NetworkChaos experiment. Chaos Mesh, an open-source chaos engineering platform, was used to inject a deliberate network delay between a catalog service and its PostgreSQL database. This simulated a real-world issue, causing slow database queries and subsequently leading to 502 errors on the front-end service. To challenge the LLMs, the provided context intentionally included misleading signals, such as query timings inflated by the artificial network round-trip time, requiring sophisticated reasoning to discern the true root cause.

The same prompt, approximately 9,800 tokens in length, was then submitted to eleven different LLMs. Each model was tasked with identifying the root cause, outlining the cause-and-effect chain, and suggesting an immediate fix.

Key Findings from Coroot’s Experiment

The results offered compelling insights into the current state of LLM reasoning for RCA:

• Frontier Models Excelled: Leading closed-source models such as Claude Opus 4.8, GPT-5.5 (presumably GPT-4.5 or a similar advanced iteration), and Gemini 3.1 Pro all successfully passed the test. They accurately identified the Chaos Mesh experiment as the root cause and correctly flagged the necessity of deleting both the experiment and its schedule to resolve the issue. This demonstrates a high level of reasoning capability when presented with well-structured, albeit challenging, context.
• Larger Open-Weight Models Competed: Many larger open-weight models largely kept pace with their closed-source counterparts, indicating that advanced reasoning is not exclusively confined to proprietary systems.
• Self-Hostable Performance: Among self-hostable models, Gemma 4 31B stood out as the only one to correctly identify the root cause. This is a significant finding for organizations prioritizing data privacy, cost control, or on-premises deployment, as it suggests viable open-source alternatives exist for critical AIOps tasks.
• Missed Diagnoses: Conversely, larger open-weight models like Qwen3.6 35B and Qwen3 Coder Next failed to pinpoint the root cause, highlighting that model size alone does not guarantee superior performance in complex reasoning tasks without sufficient fine-tuning or architectural advantages.

Industry Consensus and Broader Implications

Coroot’s findings, while not definitively settling the agentic vs. deterministic debate, strongly reinforce the industry’s growing inclination towards the latter for improved reliability and evaluation. The challenges of debugging non-deterministic agentic systems, coupled with their often higher operational costs due to extensive token usage, push many practitioners towards more controlled, deterministic workflows where the LLM performs a targeted reasoning step. Discussions on platforms like Reddit among experienced developers reflect this trend, with many reporting a shift away from fully agentic designs towards deterministic pipelines augmented with narrow LLM steps for better reliability and token cost efficiency.

This shift has profound implications for the observability market and for engineering teams worldwide:

• Focus on Context Engineering: The "real work" in AI RCA is now recognized as "preparing the right, compact context for the model before you call it." This elevates "context engineering" to a core discipline. Guidance from leading AI companies like Anthropic ("Effective Context Engineering for AI Agents") and framework providers like LangChain ("Context Engineering for Agents"), alongside observability vendors such as Mezmo ("Context Engineering for Observability"), converges on this principle: curating the smallest set of high-signal, compact context is paramount for reliable LLM-based reasoning in observability.
• R&D Shift for Observability Vendors: This means observability vendors will increasingly invest in sophisticated data correlation engines, topology mapping, causal inference algorithms, and semantic enrichment tools rather than solely focusing on LLM development. The ability to automatically identify and package relevant logs, metrics, traces, and configuration data, while filtering out noise, will become a key differentiator.
• New Skill Sets for Engineers: Site Reliability Engineers (SREs), DevOps practitioners, and incident responders will need to develop expertise not just in understanding system telemetry but also in how to structure and present that data effectively to AI models. This might involve learning prompt engineering techniques, understanding data schemas for LLM consumption, and collaborating closely with data engineers to build robust context pipelines.
• Economic Benefits: As Sivko noted, a single, short LLM call for RCA, especially when the heavy correlation work is done beforehand, costs only a few cents even on frontier models. This cost efficiency, combined with faster MTTR, translates directly into significant operational savings and reduced business impact from incidents. Studies consistently show that the average cost of downtime for businesses can range from hundreds of thousands to millions of dollars per hour, depending on the industry and scale. Improving RCA accuracy and speed directly mitigates these costs.
• Enhanced Operational Reliability: By making AI-assisted RCA more reliable and debuggable, deterministic approaches pave the way for wider adoption and greater trust in automated incident resolution. This allows human engineers to focus on more complex, novel problems that still require nuanced human judgment, rather than spending hours sifting through mountains of data for common issues.

The Future Landscape of AIOps and Observability

The journey towards fully autonomous and intelligent IT operations is ongoing. While the core reasoning of LLMs has proven remarkably capable, the current emphasis on context engineering underscores a fundamental truth in AI development: the quality of input dictates the quality of output. As observability platforms continue to evolve, we can expect to see deeper integrations between real-time data streaming, advanced causal AI engines, and context-aware LLM interfaces. This integration will aim to create a seamless flow from raw telemetry to actionable insights, further reducing the cognitive load on human operators and accelerating incident resolution. The "hardest part" of AI RCA may have shifted, but the ultimate goal remains constant: to build more resilient, observable, and self-healing systems in an increasingly complex digital world.