Rebuilding a Client-Side Photo Editing Platform: Hard Lessons in WebGPU and ONNX Runtime

The modern landscape of web development is witnessing a paradigm shift away from heavy server-side processing toward client-side execution, driven by advances in browser hardware acceleration, WebAssembly (WASM), and the WebGPU API. Recently, a developer undertook the ambitious project of modernizing a retired photo-editing web application to perform complex artificial intelligence tasks—such as object removal, background removal, and high-resolution upscaling—entirely within the user’s browser. Utilizing ONNX Runtime Web paired with WebGPU, the upgraded platform was designed to operate without remote server uploads, relying instead on local model caching after initial consent. However, the implementation revealed critical, unexpected technical roadblocks that highlight the current limitations and quirks of bringing advanced machine learning models directly to client hardware.

Project Architecture and Technological Foundations

The redesigned platform integrates several state-of-the-art computer vision models to provide a comprehensive suite of photo-editing capabilities. For object removal, the application employs MI-GAN and LaMa architectures; background removal is handled by RMBG-1.4; and image upscaling relies on Real-ESRGAN to achieve fourfold resolution enhancement.

By executing these models locally, the application eliminates server hosting costs, preserves user privacy by keeping sensitive images on the local device, and drastically reduces latency once the underlying model files are downloaded. The technical pipeline depends on ONNX Runtime Web, which acts as the abstraction layer, routing execution through WebGPU for hardware acceleration where available, with a WebAssembly fallback for older or incompatible systems. Models are downloaded only once, upon user agreement, and persistently stored locally using the browser’s Cache Storage API.

Despite the theoretical elegance of a serverless, client-side AI architecture, the practical execution exposed three major failure modes that traditional error-handling mechanisms failed to catch.

Hardware Limitations and the Trap of Model Selection

The first major hurdle encountered during development involved hardware constraints related to shader storage buffers. During the model selection phase for background removal, preliminary offline evaluations demonstrated that the BiRefNet-lite (MIT) model significantly outperformed RMBG-1.4 across a standardized test set of ten distinct images.

However, when deployed in the browser on consumer Apple hardware, the execution immediately failed during the initial session.run() call, throwing a cryptic runtime error: Too many storage buffers in shader. Current: 11, Max is 10.

An analysis of the underlying WebGPU specification on Apple’s hardware reveals a strict limit of maxStorageBuffersPerShaderStage = 10. Certain fused computational kernels generated dynamically by ONNX Runtime for the BiRefNet-lite architecture required 11 storage buffers, a demand that cannot exceed the hardware adapter’s capacity. Attempts to mitigate the issue by lowering graph optimization levels proved ineffective. Furthermore, falling back to the WebAssembly backend triggered catastrophic memory failures, resulting in a std::bad_alloc exception because the 1024×1024 transformer activations exceeded the memory limits of a standard 4GB wasm32 heap. The BEN2 model encountered identical failure patterns.

This phenomenon underscores a critical principle for modern web-based machine learning engineers: offline benchmark metrics do not guarantee browser compatibility. Developers are advised to benchmark candidate models directly inside target browsers on the weakest target hardware configurations long before finalizing quality comparisons. In this instance, the simpler convolutional model, RMBG-1.4, successfully executed in approximately 0.25 seconds via WebGPU and roughly 6 seconds via WebAssembly, whereas the theoretically superior transformer model failed to run entirely.

Silent Failures: When Execution Succeeds But Output Corrupts

The second major obstacle involved silent execution failures, where code runs to completion without generating runtime exceptions, yet produces completely corrupted results. This issue manifested clearly during the integration of the LaMa inpainting model.

When executed via WebGPU, LaMa completed its forward pass without throwing a single error. Inspection of the output tensor confirmed that the tensor shape was correct and that pixel values fell within the standard 0 to 255 range. Visually, however, the inpainted region of the image was rendered as a solid, unnatural white patch. Numerical analysis of the pixel data revealed a stark discrepancy:

webgpu  hole mean=254.3   outside mean=127.0
wasm    hole mean=107.3   outside mean=127.0

The root cause lay in the architecture of LaMa, which heavily relies on Fast Fourier Convolution operations (RFFT and IRFFT). The WebGPU execution provider implementation within the runtime produced mathematically incorrect values for these specific operations. Because no exception was thrown, the application’s built-in fallback chain—which was designed to transition from WebGPU to optimized graphs and finally to WASM only upon encountering an error—never triggered.

Consequently, the development team was forced to hardcode specific model routing, ensuring that LaMa permanently executes via the WASM backend rather than WebGPU. Additionally, this incident prompted a fundamental shift in the project’s testing methodology, establishing that automated end-to-end tests must validate actual pixel output colors rather than merely checking whether execution terminated successfully.

Evolving Browser Standards and Float16 Handling

The third unexpected challenge stemmed from rapid advancements in web standards regarding native data types, specifically floating-point precision. The Real-ESRGAN x4plus model is distributed in half-precision (fp16) format, utilizing fp16 inputs and outputs. Initially, the application logic encoded input tensors into a Uint16Array and manually decoded output values from raw half-float bit patterns.

Following updates to Google Chrome introducing native Float16Array support, the behavior of ONNX Runtime Web shifted. When native support is detected, the runtime automatically returns fp16 outputs as standard JavaScript numbers rather than raw bit representations. As a result, images processed through Real-ESRGAN suddenly rendered entirely solid black because the application attempted to decode standard floating-point numbers as if they were raw bit patterns representing zero.

To resolve this compatibility issue, the codebase had to be updated to dynamically handle both runtime environments:

const out = raw instanceof Uint16Array
  ? Float32Array.from(raw, halfBitsToNumber)
  : Float32Array.from(raw as ArrayLike<number>);

Compounding these precision issues, the same upscaling model experienced shape mismatch errors on WebGPU, specifically failing with the message Shape mismatch attempting to re-use buffer. This was ultimately resolved by explicitly pinning symbolic dimensions using freeDimensionOverrides: N: 1, H: 192, W: 192 and processing images through fixed-size sliding tiles.

Broader Industry Implications for Client-Side AI

The challenges documented in modernizing this photo-editing platform reflect broader technical hurdles facing the web development and machine learning communities. As browser vendors push to make web applications rivals to native desktop software through WebGPU, WebAssembly, and optimized tensor runtimes, developers are increasingly bridging the gap between high-performance computing and sandboxed browser environments.

However, the lack of standardization across different GPU vendors (Apple, NVIDIA, AMD, and integrated mobile graphics chips) means that WebGPU shaders can behave unpredictably. Hardware limits on storage buffers, varying support for complex mathematical operations like Fourier convolutions, and fast-moving native JavaScript type implementations create a fragile ecosystem for advanced AI workloads.

Despite these engineering friction points, the successful deployment of models like MI-GAN, RMBG-1.4, and Real-ESRGAN directly within user browsers demonstrates the immense viability of client-side machine learning. By eliminating server infrastructure costs and addressing data privacy concerns through localized processing, developers can deliver powerful, zero-latency multimedia tools directly to the end user. As ONNX Runtime Web, browser graphics APIs, and model conversion pipelines continue to mature, overcoming these foundational hurdles will pave the way for a new generation of robust, client-side intelligent web applications.