Datadog Leverages AI to Revolutionize Critical Production System, Achieving Massive Performance Gains and Cost Reductions

In a groundbreaking endeavor, Datadog, a leading monitoring and security platform for cloud applications, has successfully evolved a critical production system, the Stream Router, utilizing artificial intelligence to overcome inherent storage backend limitations and dramatically enhance performance. Arnold Wakim, an engineer at Datadog, recently detailed the intricate process, highlighting both the successes and the unexpected challenges encountered during this ambitious migration, underscoring invaluable lessons for the broader tech industry.

The Criticality of Stream Router: Datadog’s Context

Datadog’s Stream Router serves as a foundational API responsible for managing routing within the company’s extensive metrics pipeline. In an environment where real-time data ingestion and analysis are paramount, the efficiency and reliability of such a system are non-negotiable. The original iteration of the Stream Router was architected as an eventually consistent system, employing a key-value (KV) model, specifically built upon FoundationDB. This choice likely stemmed from FoundationDB’s strengths in distributed data storage, offering high availability and horizontal scalability for certain access patterns. However, as Datadog’s operational scale expanded and the complexity of its routing table grew, this architectural decision began to present significant bottlenecks.

The fundamental issue arose from the KV model’s inherent design, which, while excellent for simple data lookups, struggles with complex relational data. The Stream Router’s code was forced to reconstruct intricate relationships between domain entities within application processes. This meant pulling tens of thousands of entries into individual pod processes, effectively mimicking the functionality of a relational database in-process. This approach introduced substantial overhead, leading to operations that, in a traditional relational system, would be handled efficiently by foreign keys and optimized queries. The consequences were severe: the KV database began hitting transaction size limits, and the most demanding operations, critical for maintaining the integrity and responsiveness of the metrics pipeline, slowed dramatically. Some operations stretched to an agonizing 45 minutes, necessitated by thousands of sequential round trips to assemble and process the required data. This performance degradation posed a significant threat to Datadog’s ability to process and route customer metrics efficiently, directly impacting service quality and scalability.

From Key-Value to Relational: The Architectural Shift

Recognizing the architectural limitations, Datadog engineers embarked on a comprehensive redesign. The core of this effort involved a fundamental shift from the key-value paradigm to a relational model. This transition aimed to leverage the strengths of relational databases, particularly their ability to natively handle complex data relationships through schema definition and foreign keys. After careful analysis, PostgreSQL was selected as the new storage backend, offering robust ACID (Atomicity, Consistency, Isolation, Durability) properties, mature tooling, and strong support for complex queries. The redesign focused on meticulously crafting a new schema that accurately reflected the relationships between domain entities, allowing the database itself to enforce integrity and manage dependencies, thereby eliminating the cumbersome in-process logic previously required.

The new schema promised a more efficient and scalable foundation, but it presented an enormous challenge: refactoring the entire codebase of a critical production system to migrate from the old KV model to the new relational model. Such a task, typically labor-intensive and error-prone, would demand significant engineering resources and time. It was at this juncture that Datadog turned to artificial intelligence as a potential accelerator.

AI as an Engineering Accelerator: The Refactoring Process

Datadog’s approach to integrating AI into this critical refactoring project was highly strategic and pragmatic. Instead of deploying AI for autonomous code generation, the team leveraged advanced large language models (LLMs) like Claude and Cursor as intelligent assistants within a systematic, test-driven refactoring process. The methodology was precise: for each method requiring refactoring, engineers provided the AI models with three key inputs: the existing (old) implementation, the newly defined relational schema, and a failing test case specifically designed to validate the method’s functionality under the new schema.

The AI models would then generate a "first pass" of the refactored code. The comprehensive test suite, a cornerstone of Datadog’s engineering culture, then served as the immediate and objective arbiter of correctness. If a generated solution failed the tests, engineers would iterate, providing more context or refining prompts, until a correct solution was achieved. This human-in-the-loop, test-driven approach mitigated the risks associated with AI-generated code in a critical production environment, transforming AI from a black-box solution into a powerful, guided tool.

Pillars of Success: Modularity, Testing, and Parallel Infrastructure

Wakim identified three pivotal elements that underpinned the success of this complex migration:

1. Strong Code Modularity: The original Stream Router’s architecture exhibited robust modularity, a design principle that proved invaluable. This modularity allowed Datadog engineers to implement the new Stream Router API on PostgreSQL without necessitating widespread changes across other dependent systems. The clear separation of concerns meant that the internal storage backend could be swapped out (FoundationDB to PostgreSQL) while presenting the same external API interface, minimizing ripple effects and simplifying the deployment process.

2. A Comprehensive Test Suite: The existence of a deep and comprehensive test suite was arguably the single most critical factor. It provided an unambiguous pass/fail criterion for every AI-generated code change. This extensive suite of automated tests acted as a safety net, ensuring that even with AI assistance, the integrity and functionality of the system remained uncompromised. Without such a robust testing framework, trusting AI-generated code in a production system of this magnitude would have been an insurmountable risk. The tests provided the necessary feedback loop for iterative refinement and validation.

3. Parallel Infrastructure (Blue/Green Deployment): To ensure a seamless and risk-free transition, Datadog implemented a sophisticated parallel infrastructure. Two independent instances of the Stream Router ran side-by-side: the existing FoundationDB-backed version and the newly developed PostgreSQL-backed version. Both instances handled the same incoming requests. Client traffic was then strategically routed between these two versions using feature flags, allowing for gradual rollout and immediate rollback capabilities. A dedicated validator service, deployed in every cluster, continuously compared routing responses from both systems. This service periodically executed requests against both the old and new routers, comparing their outputs. Any divergence between the responses triggered an immediate alert to the engineering team, enabling prompt investigation and resolution of discrepancies before they could impact end-users. This "blue/green" strategy, coupled with continuous validation, was crucial for maintaining system stability and data integrity throughout the migration.

The Migration Journey: Phased Execution

The migration itself was executed in a carefully orchestrated three-phase approach, meticulously designed to manage risk and optimize the use of AI:

1. Describing Intent with AI: In the initial phase, Claude was used to articulate the precise intent and functionality of each key function within the Stream Router’s codebase. This step helped to formalize understanding, create a clear documentation baseline, and prepare the ground for subsequent refactoring. By having AI summarize the purpose of complex legacy code, engineers gained a clearer perspective and a more structured starting point for the migration.

2. Test-Driven Prompting for Fixes: The second phase involved the core refactoring. Engineers crafted highly focused prompts, supplying the AI with the expected behavior for each function and the context gleaned from the previous descriptive step. The objective was to guide Claude and Cursor in generating code that would fix the failing tests, aligning the functionality with the new relational schema. This iterative process, driven by test feedback, allowed for efficient generation and validation of new code segments.

3. Production Deployment via Blue/Green: The final phase encompassed the deployment to production. The "blue/green" approach, as detailed earlier, ensured a controlled rollout. The new PostgreSQL-backed Stream Router ran in parallel with the existing FoundationDB-backed version, with continuous comparison and validation by the dedicated service. This gradual shift, combined with robust monitoring and immediate alerting, minimized potential downtime and ensured data consistency during the transition of critical services.

Navigating the Limitations: Where AI Fell Short

Despite the significant successes, Wakim candidly acknowledged areas where the AI-assisted approach exhibited limitations, providing crucial insights into the current state of LLM capabilities in complex engineering tasks:

1. Higher-Level Prompting and Nuance: The effectiveness of higher-level, more abstract prompts proved limited. AI models performed best when provided with specific, detailed instructions and ample context. This suggests that while LLMs excel at generating code snippets based on explicit requirements, they still struggle with interpreting broader architectural goals or making nuanced design decisions without explicit human guidance.

2. The Quest for Optimal Queries: One notable challenge was the AI’s tendency to generate correct but suboptimal queries for PostgreSQL. While the generated queries consistently returned the right results, they often did so inefficiently, issuing far more round trips to the database than necessary. Niche optimizations, such as batching operations, leveraging UNNEST tricks for array processing, or employing common table expressions (CTEs) for complex query structures, frequently required direct human intervention. The AI models did not independently discover these sophisticated optimization patterns, even though they are well-documented in database literature. However, once engineers provided the optimized versions, the models demonstrated an ability to replicate these patterns in subsequent methods, indicating a capacity for learning from explicit examples, but not for independent discovery.

3. Token Consumption Challenges: The iterative nature of the refactoring process, combined with the need to provide comprehensive context, led to high token consumption. Feeding the AI models full test output dumps rather than trimmed excerpts, alongside the continuous loop of test results, code context, and schema information, contributed significantly to the computational cost. This highlights an area for future optimization in AI-assisted development workflows, focusing on more intelligent context management and prompt engineering to reduce resource usage.

Transformative Outcomes: A New Era for Stream Router

The successful migration brought about truly transformative improvements for Datadog’s Stream Router, validating the strategic investment in AI-assisted engineering:

• Dramatic Performance Gains: Operations that once took up to 45 minutes were reduced to approximately one second. Latencies, which previously ranged in the hundreds of milliseconds, plummeted to just a few milliseconds. This exponential improvement in speed fundamentally altered the system’s responsiveness and capacity.
• Resource Efficiency and Cost Savings: Data storage requirements were reduced by an astonishing 40 times. The shift to PostgreSQL and the introduction of DuckDB (likely for analytical or specialized query tasks) simplified relationship handling and significantly improved overall query efficiency. Consequently, CPU and memory usage across the system decreased, leading to substantial operational cost reductions, with database costs falling by an impressive 90%.
• Enhanced Scalability and Maintainability: The new relational architecture provides a more robust and scalable foundation for future growth. The explicit definition of relationships within the schema simplifies future development, debugging, and maintenance, reducing the cognitive load on engineers.

Strategic Insights and Broader Industry Implications

Datadog’s successful migration of its Stream Router stands as a compelling case study for the practical application of AI in complex software engineering. It offers several strategic insights and broader industry implications:

• Datadog’s AI Strategy Validated: This project validates Datadog’s strategic approach to leveraging AI as an augmentation tool rather than a replacement for human engineers. It demonstrates how AI, when carefully integrated into existing robust engineering practices, can unlock significant efficiencies and accelerate critical projects.
• A Model for Future Engineering: For other enterprises grappling with legacy systems, performance bottlenecks, or complex refactoring challenges, Datadog’s experience provides a clear blueprint. It underscores the potential for LLMs to assist in tasks traditionally considered highly manual and time-consuming, offering a pathway to modernize infrastructure more rapidly.
• The Human-AI Collaboration Imperative: The project powerfully illustrates the synergistic potential of human-AI collaboration. While AI excelled at generating initial code passes and handling repetitive transformations, human engineers remained indispensable for strategic architectural decisions, complex optimization, and critical quality assurance through rigorous testing. This hybrid approach maximizes the strengths of both human ingenuity and AI’s processing power.
• The Test Suite as Trust Anchor: Wakim’s ultimate conclusion resonates strongly: the strength and comprehensiveness of Datadog’s test suite were the paramount success factors. It was the "north star" that ultimately determined the degree of trust they could place in AI-generated code. This reinforces the long-held industry tenet that robust testing is not merely a best practice but an absolute necessity, particularly when integrating new, powerful, and potentially unpredictable tools like AI into critical systems.

In conclusion, Datadog’s journey to evolve its Stream Router showcases a meticulously planned and expertly executed engineering feat. By strategically deploying AI alongside established engineering principles like strong modularity, comprehensive testing, and blue/green deployments, Datadog not only overcame significant technical hurdles but also achieved unprecedented performance gains and cost reductions, setting a new benchmark for AI-assisted infrastructure modernization in the industry. The lessons learned, particularly regarding the need for human oversight and robust validation, will undoubtedly shape the future of AI in software development.