Netflix Overhauls Data Movement with CloudStream Initiative, Achieving 90% Faster Deployments and 70% Cost Savings

Netflix, the global streaming giant, has unveiled details of a monumental architectural pivot, dubbed CloudStream, designed to revolutionize how it moves vast quantities of data from offline data warehouses to online serving systems. This ambitious undertaking has yielded significant operational efficiencies, including a staggering 90% reduction in data deployment time and a 70% cut in associated infrastructure costs, according to Rajasekhar Ummadisetty, a software engineer on Netflix’s Online Datastores team, and Ken Kurzweil, from the Data Movement team, both within the Data Platform Organization. The transformation addresses long-standing challenges in handling terabytes of critical information, from user recommendations to personalized features, ensuring seamless and rapid delivery to millions of subscribers worldwide.

The Challenge: Bridging Offline and Online Data at Hyper-Scale

At the heart of Netflix’s operation lies an intricate data ecosystem. Offline data systems, primarily vast data warehouses, store historical user activity, content metadata, and machine learning model outputs—data not directly accessed by user devices during runtime. In contrast, online data systems are in the critical path, serving real-time information with high speed and random access requirements to devices like televisions and mobile phones. The inherent complexity of moving massive datasets between these two distinct environments presented a significant bottleneck for business-critical applications.

Previously, application teams grappled with inefficient, often custom-built batch systems, frequently relying on tools like Spark to read data from warehouses and write individual records directly into online datastores. This approach, while straightforward in concept, proved disastrous at Netflix’s scale. It led to "noisy neighbor" problems, where batch loading operations would overwhelm online applications, effectively causing self-inflicted Distributed Denial of Service (DDoS) attacks. Such contention impacted the performance and availability of user-facing services. While scaling up resources was an option, it resulted in substantial wasted capacity and cost during non-batch periods. Conversely, applying backpressure to slow down ingestion compromised data accuracy and timeliness, a critical issue for features like personalized recommendations or real-time advertising signals. For datasets exceeding terabytes, these traditional methods could take days to load, despite significant financial investment, becoming a critical blocker for high-value business applications.

"Confidence is Currency": A Guiding Philosophy for Architectural Change

Underpinning every strategic and technical decision in the CloudStream initiative was a core philosophy: "Confidence is Currency." This principle, articulated by Rajasekhar Ummadisetty, emphasizes that trust, decision-making, and progress within an organization hinge on earning the confidence of stakeholders. This confidence is built through a tripartite focus on:

• Safety: Ensuring systems can protect themselves and their users, for example, by gracefully handling errors or throttling aggressive operations.
• Observability: Providing comprehensive monitoring, measurement, and understanding of a system’s internal state and progress.
• Validation: Clearly demonstrating that a system has achieved its intended results and meets functional and non-functional requirements.

By anchoring their work in these tenets, the Netflix Data Platform team aimed to secure stakeholder buy-in and support for their ambitious goals, fostering an environment where innovation could thrive without undue risk.

The Power of Abstraction: Shielding Applications from Database Complexity

A crucial architectural foundation that enabled this transformation is Netflix’s extensive use of abstraction layers. Applications at Netflix access databases not directly, but through a layer of indirection. This strategic choice isolates application teams from the underlying complexities and constant evolution of database technologies. For instance, only 16% of Netflix’s vast Cassandra fleet is directly accessed by applications today, a number that continues to decrease. The majority of fleet interactions occur through abstractions, which collectively handle approximately 70 million queries per second (QPS), with the largest individual abstraction managing around 8 million QPS.

These abstractions solve numerous challenges for application teams:

• Stable and Uniform Interface: Teams no longer need to be experts in specific database technologies or manage their constant changes.
• Centralized Optimization: The abstraction layer optimizes for diverse access patterns, managing caching, scaling, and cluster provisioning centrally. For read-heavy use cases, dedicated caching clusters can be deployed.
• Integrated Internal Systems: Custom authentication and authorization mechanisms are centrally managed by the Data Platform, simplifying integration for all applications.
• Overcoming Database Limitations: Abstractions can intelligently address database-specific constraints, such as Cassandra’s issues with large data blobs, by implementing solutions like "chunking" to break large objects into smaller, distributed pieces.

Netflix employs various abstractions for different data types: a Key-Value (KV) abstraction for distributed HashMap-as-a-service, "entity" for documents, "time series" for temporal data, and "GraphKV" for property graphs, among others. The KV abstraction, operating as a two-level map (ID mapping to a sorted map of key-value pairs), proved particularly central to the CloudStream initiative. Its high-level architecture features stateless servers connecting to a control plane and a high-level client offering features like SLO-based hedging and near caching, abstracting away low-level protocol details.

The catalyst for widespread abstraction adoption was Cassandra’s deprecation of the Thrift protocol in version 3.0, mandating a company-wide migration to the SQL protocol. Recognizing this as a recurring challenge, the Data Platform team leveraged this moment to convince application teams to migrate to the stable KV interface, promising to handle all future disruptive technological changes. This initial API migration, focused on safety, observability, and validation, allowed teams to validate functional and non-functional requirements without initial data migration, fostering confidence through immediate feedback and easy rollback options. The subsequent cluster migration involved dual writes and a schema migration from Thrift to the KV data model, further solidifying the abstraction layer’s value.

From Stateless to Stateful: The KV Abstraction’s Evolution

The efficient movement of bulk data necessitated a fundamental shift in the KV abstraction’s architecture, transforming it from a stateless system into a stateful one. In its original stateless design, any server could handle any request, as all were identical and merely translated requests into database queries. The move to a stateful architecture meant each server would now store a unique subset of data, making it distinct.

To coordinate this complexity, Netflix introduced several new control components:

• Partition Group Manager: Servers communicate with this manager to acquire a unique lease, defining their specific data subset responsibilities.
• Central Routing Manager: This component tracks all stateful KV clusters, namespaces, and which server hosts which part of the data within those namespaces.
• Cluster Routing Layer: Within each KV cluster, this layer talks to the Central Routing Manager, caching cluster-specific routing information and relaying it to connecting applications, allowing clients to route requests directly to the correct data nodes.

This architectural shift also required extending Netflix’s deployment infrastructure to support "stateful data deploys" in addition to code deploys. The new deployment process begins by capturing application-level desires (e.g., expected reads per second, dataset size), transforming them into infrastructure-level desires (e.g., instance types, replicas, dataset to be hosted). An autoscaling group is then provisioned, and each server node acquires a unique lease from the Partition Group Manager to determine its data subset. It then fetches and loads these offline-generated data files from S3.

A critical addition to this process is a dedicated "staging stage" for validation. Unlike the previous "sawtooth pattern" where older datasets were gradually replaced, allowing a window for observation, the new stateful deployment enables near-instantaneous data switches. This rapid transition, while efficient, significantly increases the blast radius in case of data corruption. The staging phase allows application teams to spin up validator applications, connect to the staging cluster, execute requests, and build confidence before promoting the dataset to production. Once validated, routing information is updated, directing client requests to the new dataset within seconds. Older datasets are retained briefly for instant rollback capability before being destroyed.

CloudStream’s Innovative Solution: Immutable Data Paving the Way

The breakthrough for bulk data movement began by analyzing customer access patterns, revealing two primary categories: mutable (read-write) and immutable (read-only). For the latter, a theory emerged: instead of inefficiently loading data through Cassandra, converting datasets to RocksDB SSTables and directly loading them onto KV nodes would be cheaper and more performant. This approach eliminated the need for a Cassandra cluster for consensus making in read-only scenarios.

The I/O and computationally heavy work of generating these RocksDB files could be done entirely offline, removing contention from online services. New dedicated KV nodes could be stood up, pre-generating files at line rate, drastically reducing dataset deployment times. The process would then involve standing up new nodes, rerouting traffic, and decommissioning old nodes.

A Proof of Concept (POC) unequivocally validated this theory, proving the approach was fast and cost-effective, converting initial skepticism into belief. The POC’s success paved the way for a production roadmap, albeit one that initially faced the challenge of extending stateless infrastructure for stateful operations. Faced with an urgent business deadline from a critical team, Netflix opted for a "Pathfinder" approach – a temporary, intentionally planned-for-migration solution. This allowed them to leverage existing high-level clients and deploy a hardened version of the POC in a limited capacity, unblocking the business while gathering crucial learnings. The Pathfinder was fast and well-understood, allowing Netflix to benchmark performance and iterate on the production KV implementation until it matched or exceeded the Pathfinder’s capabilities, benefiting all KV use cases. It also taught them how to convert, store (in S3), and deploy datasets on demand.

Tangible Business Impact: Millions Saved, Innovation Accelerated

The commitment to this architectural transformation and cross-functional collaboration delivered substantial business value across Netflix. Several critical use cases, previously struggling with traditional data movement, were revolutionized:

• Massive Datasets: For one use case involving immense datasets, traditional methods could not meet Service Level Objective (SLO) requirements, with load times extending to days and incurring annual infrastructure costs in the millions. CloudStream now enables loading these datasets in under 40 minutes and switching them over in seconds, a 99% reduction in deployment time and 70% in cost savings.
• High-Velocity Versioned Feature Data: Another use case, characterized by smaller but extremely high-frequency versioned feature data, saw load times reduced to less than 10 minutes, achieving 90% operational savings. Crucially, this solution also introduced the previously unavailable capability to roll back and roll forward between dataset versions seamlessly.
• Platform Development: Perhaps one of the most significant wins was enabling platform development. An application team successfully built a robust machine learning platform atop the CloudStream solution. This platform now empowers multiple teams to rapidly deploy and consume custom ML features, transforming the initiative from a mere technical fix into a true accelerator for business innovation at Netflix.

These successes unlocked major business investments and formalized the architectural principles underpinning CloudStream.

Generalizing the Architecture: The Capture, Conversion, Deployment Framework

During the development of CloudStream, the team generalized their methodology into a reusable and repeatable framework: Capture, Conversion, Deployment. This modular approach decouples data movement into three distinct phases, allowing different teams to work independently and optimize each stage:

• Capture: This phase creates and stores an immutable artifact of the dataset, representing its original state at the time of capture. The goal is an unalterable artifact for subsequent processing. Captures can be used directly for conversion or combined to generate efficient deltas.
• Conversion: The captured artifact is transformed into a format optimized for deployment. For KV, this initially involved RocksDB SSTables, but can be any target-specific format. This tailoring ensures optimal performance for the specific datastore.
• Deployment: This phase orchestrates the transfer of staged artifacts to the datastore and brings the datasets online deterministically. This includes provisioning new clusters, loading data, and managing traffic switching. The independence of this phase from capture and conversion is critical, enabling straightforward rollbacks and roll forwards to previous datasets.

This framework allows continuous dataset updates, triggered or scheduled captures, and conversions into formats like RocksDB SSTables or, more recently, Cassandra SSTables (currently under testing). The modularity extends to future enhancements, allowing new conversion phases for different datastores without disrupting existing processes. Optimizations in one component, such as faster dataset capture, yield benefits across the entire ecosystem. This deterministic artifact deployment acts as a reliable safety net, enhancing confidence and safety during data movement operations and fostering collaboration among diverse teams like data movement, KV, and UX.

The Road Ahead: Expanding to Mutable Data and Beyond

Looking to the near future, Netflix is actively applying the learnings from immutable datasets to the more complex domain of mutable datasets. The Capture phase will remain consistent, but the Conversion phase will now produce Cassandra SSTables, leveraging the Apache Cassandra analytics project. The Deployment phase will involve loading and generating these files into a live Cassandra cluster actively serving traffic. This process utilizes custom orchestration combined with Cassandra’s built-in file loading, compaction, and conflict resolution mechanisms.

Ken Kurzweil expressed excitement about the evolving ecosystem of data movement tools and techniques. Netflix has revolutionized its ability to read and write Cassandra SSTables, from exporting them to S3 for backups to importing them into live Cassandra clusters. Furthermore, the underlying capability now exists to write to any datastore format, enabling seamless migrations between different database technologies behind the abstraction layer—for example, switching from Cassandra to DynamoDB if cost or performance trade-offs warrant such a change.

Key Principles for Tech Organizations

The Netflix team distilled several key takeaways from their CloudStream journey, offering valuable insights for other organizations:

• Confidence is Currency: Prioritize safety, observability, and validation as core architectural principles to build stakeholder trust.
• Abstraction is a Superpower: Leverage abstraction and indirection to perform large, seamless migrations and protect applications from underlying technology changes.
• Analyze and Exploit Access Patterns: Deeply understand stakeholder use cases and data access patterns, as this is where true innovation is found.
• A POC is Worth 100 Meetings: Use Proof of Concepts to validate ideas, overcome skepticism, and convince stakeholders effectively.
• Pathfinders for Accelerated Learning: Intentionally plan temporary "Pathfinder" solutions to unblock critical business needs, learn "unknown unknowns," and gather insights while providing immediate value.
• Leverage Architectural Patterns: Employ generalized architectural patterns like Capture, Conversion, Deployment to enable modular development, independent optimization, and faster team collaboration.

The Netflix CloudStream initiative stands as a testament to the power of strategic architectural pivots, meticulous execution, and a commitment to core engineering principles in addressing the challenges of hyper-scale data management. The resulting efficiencies and capabilities underscore Netflix’s continued innovation in delivering a world-class streaming experience.