At the edge, the number that matters is memory – not throughput (specially in Ramageddon)

In the world of industrial Internet of Things (IIoT), edge computing, and connected vehicle telemetry, stream-processing benchmarks have long suffered from a misplaced obsession: peak throughput on enterprise-grade servers. For years, software engineers and enterprise architects have relied on performance metrics generated in pristine lab environments boasting dozens of CPU cores and abundant memory pools. However, these figures bear little relevance to the operational reality of edge devices.

In field deployments—such as industrial automation gateways, ESPHome smart home hubs, vehicle infotainment head-units, and electric vehicle (EV) charging stations—computing resources are severely constrained. Systems typically operate on a single CPU core or two, backed by a few hundred megabytes of free memory. Inputs arrive primarily over the MQTT protocol, and traffic patterns are notoriously volatile. Fleets of devices regularly reconnect en masse following network outages, EV chargers initiate simultaneous session logs, and remote sensors periodically flush buffered telemetry data all at once.

Under these punishing conditions, traditional stream processors frequently falter, victims of unbounded memory consumption and uncontrolled queue expansion. Addressing this systemic vulnerability, engineers at I-Dacs Labs have developed rekuiper, a high-performance stream-processing engine written in Rust designed specifically to mirror the surface area of LF Edge eKuiper while fundamentally reengineering memory management beneath the hood.

The Engineering Imperative: Bounded Memory at the Edge

Traditional stream-processing pipelines often treat memory as an elastic resource, expanding dynamically to accommodate incoming message spikes. While effective in cloud environments with virtually unlimited RAM, this architectural philosophy proves catastrophic on edge hardware capped at 1 GB of memory or less. When a fleet of thousands of devices reconnects simultaneously, naive stream processors buffer incoming rows indiscriminately, triggering Out-Of-Memory (OOM) crashes or forcing operating systems to invoke swap space, which introduces debilitating latency spikes.

To evaluate how modern stream processors handle these constraints, the development team behind rekuiper constructed a rigorous, highly controlled benchmarking harness. Rather than relying on throughput metrics alone—which can easily mask data loss, message duplication, or structural corruption—the testing framework enforced message-by-message output verification.

Four distinct engines were subjected to the evaluation: rekuiper v0.425-beta, LF Edge eKuiper 2.4.1, Telegraf 1.40.0, and Redpanda Connect 4.109.0 (formerly known as Benthos). Apache Flink was deliberately excluded from the test suite because neither Flink 2.x nor Apache Bahir provides a native MQTT connector. Introducing a custom source or an intermediary Kafka bridge would have fundamentally altered the ingest path, invalidating direct comparisons.

The benchmark environment was strictly constrained. Each engine execution was pinned inside a container limited to exactly one CPU core, 1 GB of memory, and zero swap space via explicit cgroup directives (--cpuset-cpus=2 --cpus=1 --memory=1g --memory-swap=1g). A dedicated Mosquitto MQTT broker operated on separate CPU cores with generous queue limits to ensure the message broker itself never became the operational bottleneck. An open-loop Rust-based load generator, utilizing standard libraries and operating on MQTT 3.1.1 at QoS 0, fed every engine according to an identical, deterministic schedule.

Five Real-World MQTT Workloads

To simulate authentic industrial and commercial edge deployments, the benchmark suite incorporated five distinct workloads, each testing specific operational capabilities:

1. W1 (Telemetry Filter): A straightforward stateless filtering pipeline evaluating raw incoming JSON sensor payloads against numerical thresholds, routing valid messages to persistent storage.
2. W2 (Per-Device Windows): A stateful workload executing time-based sliding and tumbling windows grouped by unique device identifiers, calculating metrics such as counts, averages, minimums, and maximums.
3. W3 (ESPHome States): A wide-schema workload processing diverse, heterogeneous state publications typical of smart home environments and IoT automation controllers.
4. W4 (Vehicle Windows): A high-frequency telemetry workload simulating vehicular telematics fleets transmitting rapid location, speed, and diagnostic parameters requiring continuous windowed aggregation.
5. W5 (Charger Sessions): A complex stateful workflow managing EV charging session aggregation, tracking energy consumption metrics, session durations, and billing triggers across concurrent connection pools.

For every engine, workload, and ingestion rate—tested at 5,000, 20,000, 50,000, and 100,000 messages per second—the testing protocol followed a strict sequence: warm-up until subscriptions were provably active, a fixed 30-second sustained load phase, complete draining of the output sink files, and rigorous cryptographic or line-by-line verification of the resulting data against expected outputs. A test phase was deemed successful only if output completeness matched input validity without data loss or record corruption.

Architectural Innovations: Why rekuiper Maintains Flat Memory

The core architectural differentiator of rekuiper lies not merely in its choice of implementation language (Rust), but in three deliberate structural design choices intended to enforce predictable resource utilization under extreme duress.

First, rekuiper implements bounded queues with strict backpressure. Sources publish records into an in-process stream bus utilizing fixed-size per-subscriber queues capped at 4,096 records. Admission relies on a reserve-then-commit protocol: batches must successfully reserve capacity across every subscriber queue simultaneously before admission is finalized. Consequently, data is either fully propagated or rejected outright, preventing partial deliveries. Slow rules exert backpressure directly onto the data source rather than silently dropping records or allowing memory buffers to swell indefinitely.

Second, rekuiper introduces incremental window aggregation operating at $O(groups)$ complexity rather than $O(messages)$ complexity. Traditional stream engines buffer every individual row falling within a temporal window and compute aggregations upon the window trigger. Under high traffic or fleet reconnection scenarios, memory usage spirals upward linearly with message volume. In contrast, rekuiper maintains a single accumulator per grouping key per aggregate function. Window memory consumption thus scales strictly with the number of active devices in the fleet rather than the aggregate volume of incoming messages. For complex operations requiring raw row access—such as joins or specific HAVING clauses—the system falls back to a verified buffered evaluator, validated through automated unit tests.

Third, the engine features an offline sink cache with disk spilling capabilities. Borrowing configuration paradigms from eKuiper (enableCache, memoryCacheThreshold, and maxDiskCache), rekuiper handles intermittent network uplinks—such as vehicles entering tunnels or remote industrial sites experiencing cellular dropouts—by buffering recoverable send failures in memory up to a defined threshold, subsequently spilling overflow data to structured disk pages. Oldest records are pruned only when absolute disk quotas are exhausted, with dropped records explicitly counted rather than silently discarded.

Empirical Findings: Unprecedented Headroom and Efficiency

The benchmark results revealed stark operational divergences between the tested engines when subjected to constrained hardware limits.

Across all five workloads, rekuiper successfully processed complete, verified output up to 100,000 messages per second on a single CPU core, which represented the upper ceiling of the tested generation range. At 100,000 messages per second on wide schemas, CPU utilization hovered at approximately 96.6%, while windowed workloads utilized between 78% and 85% of the single core, indicating remaining operational headroom.

By contrast, competing engines encountered structural limitations much earlier in the scaling curve. LF Edge eKuiper 2.4.1 delivered robust, complete output up to 20,000 messages per second across all workloads, but experienced output degradation or ingestion loss at higher rates. Telegraf 1.40.0 successfully managed 50,000 messages per second on stateless workloads but failed to produce complete per-device windowed outputs consistently. Redpanda Connect achieved 20,000 messages per second on stateless pipelines and 5,000 messages per second on windowed configurations.

The most striking metric, however, centered on memory utilization. Measured at 20,000 messages per second—the highest ingestion rate where all engines could be directly compared—the memory gap was profound. On windowed workloads (W2, W4, and W5), eKuiper’s heap consumption expanded to between 536 MB and 886 MB, while Redpanda Connect’s system-window implementation routinely breached the 1 GB hardware ceiling, resulting in container termination. Meanwhile, rekuiper maintained a remarkably flat memory footprint ranging between 5 MB and 10 MB across every rate and workload. This represents a two-order-of-magnitude reduction in heap allocation under identical operational conditions.

Furthermore, CPU efficiency favored the Rust-based implementation. At 20,000 messages per second, rekuiper consumed between 44% and 49% of a single CPU core, whereas eKuiper utilized between 86% and 99%. This efficiency margin ensures that edge gateways running ancillary local services maintain adequate processing headroom during traffic spikes.

Methodological Rigor and the Discovery of Internal Defects

The development team emphasized that the credibility of their findings rested heavily on the stringency of the verification harness. During preliminary benchmark iterations, a recurring anomaly manifested as a consistent 15% data shortfall on workload W2, alongside total processing collapse at 100,000 messages per second. Initial assumptions attributed the issue to processing overload.

Subsequent investigation revealed a structural correctness defect within rekuiper’s internal window evaluation logic. The time-window trigger was erroneously collapsing entire multi-group windows into a single aggregated record, ignoring GROUP BY partitioning and bypassing WHERE filter clauses. Standard unit tests failed to capture the defect because they typically evaluated isolated single-group scenarios. Only the comprehensive, message-by-message output verification harness exposed the semantic mismatch. Recognizing and rectifying this defect led directly to the formulation of the incremental aggregation engine deployed in the final benchmark runs.

The research also highlighted the critical influence of underlying I/O subsystems. Initial test configurations utilizing virtualized bind mounts for output sink logging artificially penalized engines like Telegraf and Redpanda Connect, which perform unbuffered or individual file writes per message. Transitioning output sinks to native local filesystems significantly altered performance outcomes, reinforcing the broader engineering principle that edge benchmarking methodologies must explicitly document storage I/O parameters to ensure reproducibility.

Implications for Edge Architecture and Future Outlook

The empirical data generated by the rekuiper evaluation challenges prevailing assumptions regarding language runtimes and stream-processing design. While memory-managed languages such as Go offer exceptional developer velocity and ecosystem maturity, garbage collection pressures and naive window-buffering strategies introduce severe liability margins in resource-constrained environments.

By prioritizing bounded memory architectures, incremental aggregation mathematics, and rigorous output verification over raw, unverified throughput figures, the engineering community gains a clearer roadmap for building resilient distributed infrastructure. As industrial automation, smart grid deployment, and connected vehicle fleets continue to expand the volume of edge-generated telemetry, the operational viability of stream processing will increasingly depend on predictable resource scaling rather than peak performance metrics achievable only under idealized server conditions.

rekuiper v0.425-beta is distributed under dual licensing terms via the MIT and Apache-2.0 licenses. Complete replication packages, including orchestrator scripts, load generator configurations, raw telemetry logs, and per-second container resource metrics, have been published to the official public repository to encourage independent verification, auditing, and stress testing across diverse hardware architectures, including ARM-based edge gateways.