AWS Details How One Customer Scaled to One Million Lambda Functions

AWS has recently detailed how ProGlove, a prominent manufacturer of industrial wearables, successfully engineered and scaled its Software-as-a-Service (SaaS) platform to operate more than one million AWS Lambda functions. This monumental achievement is distributed across thousands of dedicated customer accounts, underscoring a significant milestone in serverless architecture and multi-tenant cloud operations. The insights, shared in an AWS Architecture Blog post, highlight ProGlove’s innovative approach, which hinged on a one-account-per-tenant model, extensive automation leveraging CloudFormation StackSets, and aggressive scale-to-zero policies to maintain remarkably low idle costs—reportedly below one US dollar per month per account.

The Industrial Innovator: ProGlove’s Mission and Platform

ProGlove specializes in developing smart wearable solutions for industrial environments, primarily focusing on improving efficiency and accuracy in manufacturing, logistics, and warehousing operations. Their flagship products include smart gloves equipped with integrated scanners, designed to streamline workflows by enabling hands-free data capture. The SaaS platform in question serves as the backbone for these devices, ingesting critical scanner readings, processing them, and emitting valuable business events for analytics. This platform is not merely a data conduit; it’s an intelligent system that provides real-time insights, optimizes operational flows, and enhances worker productivity across a diverse client base ranging from small businesses to large enterprises. The inherent nature of their business—handling sensitive operational data for numerous independent clients—made security, isolation, and reliable performance paramount.

Architectural Cornerstone: The One-Account-Per-Tenant Model

A cornerstone of ProGlove’s architecture, and arguably the most impactful decision outlined by AWS, was the adoption of a one-account-per-tenant model. In the realm of SaaS multi-tenancy, providers often choose between several approaches: a pooled model where all tenants share the same application instances and database, a siloed model where tenants have dedicated application instances but share a database, or a fully isolated model where each tenant receives a completely separate infrastructure stack, often within its own cloud account. ProGlove opted for this latter, more stringent isolation strategy.

While this choice inherently introduces additional operational overhead compared to a more consolidated architecture, ProGlove strategically traded this for several compelling advantages. Firstly, it established robust security boundaries, ensuring that any compromise in one tenant’s environment would not directly impact another. This "blast radius" containment is critical for enterprises handling proprietary and sensitive operational data. Secondly, it provided independent service quotas for each customer. AWS services, including Lambda, impose soft and hard limits on resource usage per account. By isolating each tenant, ProGlove effectively sidestepped potential quota contention issues that could arise in a shared-account model, ensuring consistent performance for all customers regardless of the activities of others. Finally, this model offered crystal-clear per-customer cost attribution. Billing for each account directly reflects that tenant’s resource consumption, simplifying cost management, chargebacks, and financial forecasting, which is particularly beneficial for a growing SaaS business.

AWS acknowledges that this decision, while strategic, began to introduce operational friction once the platform expanded beyond approximately 50 accounts. This threshold often represents a turning point where manual processes become unsustainable, necessitating a pivot towards robust automation.

Overcoming Operational Friction: The Rise of Automation

Early in the platform’s lifecycle, the provisioning and management of individual customer accounts were likely handled through manual processes or rudimentary scripts. This approach, common in the nascent stages of many SaaS platforms, quickly became a bottleneck as ProGlove’s customer base grew. Manual provisioning slowed down the release cycle for new features and updates, increased the risk of human error, and, crucially, exposed the platform to service-quota ceilings more frequently as tenants scaled.

Recognizing these inefficiencies, ProGlove’s engineering team embarked on a comprehensive automation initiative. Their solution involved a sophisticated orchestration of several core AWS services: AWS Organizations, AWS Step Functions, and AWS CloudFormation StackSets.

• AWS Organizations provided the foundational framework, allowing ProGlove to centrally manage and govern multiple AWS accounts, grouping them logically and applying policies consistently.
• AWS Step Functions served as the workflow orchestrator, defining and executing the multi-step processes required for creating, updating, and managing customer environments. This serverless workflow service allowed engineers to build resilient and scalable state machines that could handle complex sequences of operations.
• AWS CloudFormation StackSets emerged as the linchpin for deploying and managing infrastructure as code (IaC) across these numerous accounts. CloudFormation StackSets extend the power of CloudFormation by enabling the deployment of a common set of AWS resources (a "stack") to multiple target accounts and regions from a single CloudFormation template.

This integrated workflow transformed their operational paradigm. Instead of manual intervention for each account, a single pipeline could now initiate the creation and update of every customer environment. This move dramatically improved consistency, reduced deployment times, and mitigated the risk of configuration drift between accounts.

Engineering Collaboration and Throughput Breakthroughs

The ambition of managing thousands of accounts, each hosting a complex set of microservices, pushed the boundaries of existing tools. The initial throughput of CloudFormation StackSets, while a vast improvement over manual methods, still faced limitations when fanning out changes across such an immense scale. This challenge led to a critical collaboration between ProGlove’s engineers and AWS service engineers. Such partnerships are common when companies operate at the extreme edges of cloud scale, often leading to improvements that benefit the broader AWS customer base. Through this collaboration, specific enhancements were made to StackSet mechanisms, significantly increasing their throughput. This breakthrough allowed ProGlove to propagate changes across thousands of accounts, collectively hosting over one million Lambda functions, with the required speed and reliability. This not only validated their architectural choice but also demonstrated the extensibility and adaptability of AWS’s core infrastructure services.

Navigating Scaling Hurdles: From Quotas to Self-Inflicted DDoS

Beyond the initial operational friction, ProGlove encountered several other scaling challenges inherent to distributed serverless architectures:

1. Microservices Architecture within Accounts: Inside each customer account, ProGlove deployed a lean microservices architecture. This typically comprised five to fifteen AWS Lambda functions, coordinated by an AWS Step Functions state machine. This setup was responsible for ingesting scanner readings, persisting them efficiently in Amazon DynamoDB (a NoSQL database known for its scalability), and emitting business events to a shared Amazon EventBridge bus. These events would then be consumed by ProGlove’s centralized analytics plane, demonstrating a well-architected, event-driven approach.

2. Service Quotas: As mentioned, the one-account-per-tenant model largely mitigated quota contention. However, even with isolation, individual accounts or specific regions could approach service limits if not carefully monitored and managed. The automation pipeline was crucial here, not only for deployment but also for ensuring configurations remained within permissible limits or for proactively requesting quota increases when necessary.

   

3. The "Self-Inflicted DDoS": One particularly insightful challenge emerged from their early scheduling practices. ProGlove initially used cron-style schedules to trigger functions across all accounts at identical times. While seemingly benign, this synchronized execution across thousands of accounts effectively created a "self-inflicted Distributed Denial of Service (DDoS)" attack on their own infrastructure and shared AWS services. Thousands of Lambda functions, all attempting to execute simultaneously, created massive spikes in regional load, potentially impacting performance and increasing costs.

   The solution to this problem was elegant and highly effective: replacing rigid timers with jittered execution windows and event-driven triggers.

   • Jittered Execution: Instead of all functions triggering at precisely 00:00 UTC, for example, they would be configured to trigger within a randomized window (e.g., between 00:00 and 00:15 UTC). This subtle randomization smoothed out the load spikes, distributing the computational burden more evenly over time.
   • Event-Driven Triggers: Shifting to an event-driven paradigm further reduced reliance on fixed schedules. Functions would execute only when a specific event occurred (e.g., a batch of data became available, a user action was performed), ensuring resources were consumed only when actively needed. This strategy proved so effective that it allowed ProGlove to manage regional load efficiently without resorting to more expensive solutions like reserved concurrency or provisioned capacity for their Lambda functions, which would have negated some of the cost benefits of serverless.

Mastering Observability and Cost Efficiency

The principle of "you can’t optimize what you can’t measure" is especially true in large-scale cloud environments. However, observability itself can become a significant cost center. Each of the million-plus Lambda functions emitted logs and metrics, a critical requirement for debugging, performance monitoring, and security auditing. Multiplied across thousands of accounts, the sheer volume of this data threatened to dominate ProGlove’s overall cloud bill.

ProGlove addressed this proactively with several key strategies:

• Consolidated High-Priority Failures: Instead of sending all logs from all accounts to a central logging system, which would incur massive data transfer and storage costs, ProGlove focused on consolidating high-priority failures. This involved routing critical errors and dead-letter queue messages from Amazon SQS (Simple Queue Service) to a central location for immediate attention. This "exception-based" logging strategy ensured that engineers had visibility into critical issues without drowning in irrelevant data.
• Removal of Unused Resources: A common source of cloud waste is dormant or unused resources. ProGlove implemented processes to identify and remove unused Amazon SQS queues and other resources that were no longer active. This continuous housekeeping significantly contributed to cost reduction.
• Aggressive Scale-to-Zero Policies: As highlighted earlier, Lambda’s inherent pay-per-execution model, combined with ProGlove’s architectural decisions, meant that idle resources were virtually eliminated. When a function isn’t executing, it costs nothing. This, along with the other optimization efforts, successfully brought the per-account idle spend down to less than one US dollar per month, a testament to meticulous cost management.

Industry Perspective: Serverless Adoption and Best Practices

ProGlove’s journey echoes similar operational concerns and best practices observed across the industry. Capital One, a financial services giant, has extensively documented its serverless journey, emphasizing the necessity of standardizing deployment, observability, and governance practices to operate Lambda workloads consistently across large organizations. Their experience reinforces the idea that technical scale mandates operational standardization—a lesson ProGlove clearly embraced.

Similarly, DoorDash’s serverless migration case study highlights the importance of disciplined automation in managing over ten million daily API calls without overrunning concurrency limits. While the specific tactics might differ based on business context and scale, the common thread across these high-profile serverless adopters is clear: proactive cost and quota visibility, coupled with robust automation, drives architectural choices long before capacity crunches become critical problems. These companies demonstrate that serverless architectures, when managed correctly, are not just about scalability but also about efficiency and resilience.

The AWS blog post also candidly acknowledges that automation, while transformative, involved its own set of compromises. CloudFormation StackSet rollouts, for instance, remained inherently slower than single-account CloudFormation deployments due to the distributed nature of the operations. Furthermore, centralizing logs, while cost-effective, necessarily reduced some of the granular, per-tenant observability that might be available in a fully decentralized logging strategy. However, ProGlove’s team deemed these compromises acceptable, given the significant improvements achieved in customer isolation, transparent cost visibility, and overall operational consistency across their vast fleet of accounts.

Statements from Leadership: A Collaborative Success Story

While specific direct quotes were not provided in the original summary, based on the context, one can infer the sentiments of those involved.

An AWS spokesperson might state, "ProGlove’s innovative application of AWS serverless technologies, particularly their multi-account strategy and sophisticated automation with CloudFormation StackSets, serves as a powerful testament to the scalability and flexibility of our cloud platform. Their journey provides an invaluable blueprint for other enterprises seeking to build highly secure, cost-efficient, and resilient SaaS solutions at an unprecedented scale."

A ProGlove engineering lead could be imagined to say, "Our commitment to providing unparalleled security and performance for our industrial customers drove our architectural decisions, particularly the one-account-per-tenant model. While it introduced initial operational complexities, the deep collaboration with AWS service engineers and our relentless focus on automation, particularly with CloudFormation StackSets, allowed us to overcome these hurdles, enabling us to scale efficiently to over a million Lambda functions while keeping our operational costs incredibly low. This success underscores the power of serverless and a well-thought-out cloud strategy."

Implications for the Cloud-Native Landscape

ProGlove’s experience offers several profound implications for the broader cloud-native and serverless landscape:

1. Validation of Multi-Account Tenancy: For SaaS providers with stringent security, compliance, or performance isolation requirements, the one-account-per-tenant model, despite its complexities, is validated as a viable and highly effective strategy when coupled with advanced automation.
2. Serverless for Extreme Scale: This case study unequivocally demonstrates that AWS Lambda and other serverless components are not just for small, bursty workloads but are capable of handling extreme enterprise-level scale and complexity, processing millions of functions across a distributed environment.
3. The Imperative of Infrastructure as Code (IaC): The transition from manual provisioning to CloudFormation StackSets highlights IaC as non-negotiable for managing modern, large-scale cloud infrastructure. It ensures consistency, repeatability, and agility.
4. FinOps and Cost Optimization: ProGlove’s aggressive focus on cost efficiency, particularly the scale-to-zero policies and smart observability strategies, underscores the growing importance of FinOps—the operationalization of cloud financial management—as a critical discipline for all cloud users.
5. Event-Driven Architectures (EDA): The shift from cron-style schedules to jittered execution and event-driven triggers reinforces the resilience and efficiency benefits of EDAs, particularly in preventing self-inflicted performance issues.
6. The Value of Collaboration: The success achieved through collaboration with AWS service engineers emphasizes the value of leveraging cloud provider expertise when pushing the boundaries of technology.

Conclusion: A Blueprint for Future SaaS Providers

ProGlove’s remarkable achievement in scaling its SaaS platform to over one million AWS Lambda functions provides a compelling blueprint for other enterprises navigating the complexities of multi-tenant serverless architectures. Their journey illustrates a clear pattern for success: start with a tenancy model that enforces hard boundaries, automate every infrastructure action from the earliest possible stage, and treat observability spend as a first-class scaling constraint.

By meticulously planning their architecture, embracing advanced automation tools like CloudFormation StackSets, and proactively addressing challenges such as synchronized function execution and observability costs, ProGlove has not only built a highly scalable and secure platform but has also set a new standard for operational efficiency in the serverless era. Their story is a powerful reminder that while cloud native offers immense power, it demands discipline, foresight, and continuous optimization to unlock its full potential.