Engineering AI for Creativity and Curiosity on Mobile: From Foundational Models to Product Innovation

Google is increasingly at the forefront of integrating sophisticated artificial intelligence into everyday mobile experiences, transforming how users interact with their devices. A recent presentation by Bhavuk Jain, a key engineer behind several of these innovations, shed light on the intricate process of productizing advanced AI models for mass consumer use. The discussion focused on two flagship features: AI Wallpapers, which leverages generative AI for personalized digital art, and Circle to Search, a multimodal understanding AI designed to provide instant, contextual information directly from a user’s screen. Jain’s insights detailed the architectural journey from raw foundational models to robust, user-facing products, emphasizing critical steps like post-training, fine-tuning, retrieval, grounding, and the non-negotiable implementation of comprehensive safety guardrails.

From Abstract Models to Tangible Products: The AI Engineering Blueprint

Modern AI product development rarely starts from a blank slate. Instead, it typically commences with vast, general-purpose foundational models that, while powerful, are inherently unpredictable and unconstrained. Transforming these models into reliable consumer features requires a structured, multi-stage architectural approach, as outlined by Jain. This process ensures alignment with human preferences, specialization for specific use cases, factual accuracy, and safe, scalable deployment.

The first critical step is post-training, often likened to teaching a brilliant but raw student manners and safety. This phase involves aligning the foundational model with human preferences, desired styles, and essential safety rules. Two primary families of techniques dominate this stage: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). While RLHF offers flexibility, it is complex, requiring a base model, a separate reward model, and an intricate reinforcement learning loop. DPO, conversely, is simpler and more scalable, operating directly on preferred versus rejected outputs through a supervised learning objective, bypassing the need for a separate reward model. Regardless of the chosen method, rigorous evaluation using clear, measurable criteria (such as format, quality, completeness, and accuracy) is paramount, especially given the subjective nature of "good behavior" in AI outputs. This alignment phase establishes the foundational behavioral framework before further specialization.

Following post-training, fine-tuning takes the aligned generalist model and specializes it into an expert for specific tasks, domains, or even individual users. Standard supervised fine-tuning, or "task and instruction tuning," involves training the model on large datasets of input-output pairs (e.g., questions and answers) to teach new, specific behaviors. However, the modern "sweet spot" often lies in Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA, QLoRA, and adapters. These methods allow for the training of tiny, specialized modules that operate alongside a largely frozen, large model, dramatically reducing training costs and time. It’s also recognized that not all problems require fine-tuning; if the issue is a lack of fresh, factual, or private knowledge, the solution lies in grounding.

Retrieval and grounding have become increasingly vital in combating AI hallucinations. This layer connects the model to external sources of truth, such as structured knowledge bases, real-time web results, or on-device context. By doing so, it ensures the system is accurate, factual, and up-to-date, allowing the model to respond with verified information rather than generating plausible but incorrect guesses. This is where modern AI systems integrate search, embeddings, Retrieval-Augmented Generation (RAG), or tool-use mechanisms.

The final stage encompasses inference, guardrails, and deployment. Even fine-tuned models can be unpredictable, making a dedicated, multilayered safety architecture—known as guardrails—non-negotiable for shipping any AI product. Guardrails constrain what the model can see, say, and do. These include:

• Data Guardrails: Filtering and redacting sensitive or toxic content from training and evaluation data.
• Model Guardrails: Using safety-tuned base models, strong system prompts, and separate safety classifiers to reshape model output.
• Runtime Guardrails: Real-time inspection of inputs and outputs during inference to detect PII, toxicity, or "jailbreak" attempts, alongside strict control over external tool or API calls.
• Governance and UX: Human review flows, logging, audits, and user-facing controls like reporting and feedback mechanisms.
  These layers transform an impressive lab experiment into a safe, reliable, and predictable feature.

Deployment is the moment of truth, shifting the focus from model quality to infrastructure stability. It involves unified multimodal serving, capable of handling diverse input types (text, images, audio, video) through a single API or pipeline. Engineering battles are fought over latency, cost, and scaling, requiring sophisticated techniques like batching, caching, and autoscaling to keep operations efficient and responses fast, all while managing expensive GPU and TPU resources. Observability through rich logging and metrics is crucial for tracking performance, errors, and usage in real-time. This robust infrastructure is the bridge enabling millions of users to interact with AI products daily without incident.

Case Study 1: AI Wallpapers – The Generative Canvas for Creativity

The AI Wallpapers feature, launched approximately two years ago, exemplifies the generative phase of AI engineering, where the primary optimization is for creativity and personalization. The choice of wallpapers as the initial canvas was strategic for several reasons. Firstly, it offered an immediate user benefit: deep personalization. This aligned perfectly with Android’s "Material You" design philosophy, which dynamically adapts the phone’s interface colors to match the user’s wallpaper, creating a uniquely personal experience. An AI-powered wallpaper creator was a natural extension of this vision.

Secondly, there was a clear and massive user need. Data consistently showed that changing wallpapers was the most popular way users customized their phones, often involving significant time spent scrolling through galleries for the "perfect" image. Thirdly, generative AI provided the technological breakthrough to solve this problem at an unprecedented level. Users were no longer confined to static galleries; they could become artists, creating truly one-of-a-kind digital art.

Developing AI Wallpapers presented four core challenges:

1. UX Challenge (Prompt Engineering): Traditional prompt engineering is complex, requiring users to understand esoteric terms like "cinematic lighting" or "ethereal watercolor." The goal was to simplify this into an intuitive, fun, and fast mobile UI.
2. Quality Challenge: Generative models lack inherent artistic taste. They can produce technically correct but artistically messy or "weird" images. The task was to achieve consistent artistic control that users would find appealing for their device backgrounds.
3. Safety Challenge: For a feature deployed on millions of consumer devices, preventing the generation of inappropriate, harmful, or copyrighted content was a day-one, non-negotiable requirement.
4. Viability Challenge (Cost): Early server-based generative models were powerful but incredibly expensive to run. As adoption grew, operational costs became a significant concern, threatening long-term viability.

To address the UX challenge, the team made a "non-obvious bet" against the prevailing wisdom of maximum user freedom. Instead of an open text box, they opted for a hybrid model of guided prompt building. This approach offered curated concepts and creative building blocks, guiding users to construct effective prompts while still allowing for unique ideas. This decision directly influenced engineering, enabling the definition of narrow, measurable quality criteria (format, conciseness, accuracy) and simplifying the implementation of reliable guardrails. User experience research played a critical role in validating this approach with real users.

Crucially, with restriction came the responsibility to invest heavily in delight. This included subtle motion design in the UI, such as wavy or horizontally scrolling text in suggestion chips, making the interface feel alive. Delight also extended to handling failures gracefully; instead of harsh error messages, users might encounter playful visuals like a T-Rex trying to paint, transforming potential frustration into a whimsical experience. This focus on a guided, delightful experience also had significant safety implications, directly reducing the likelihood and cost of catastrophic safety failures (e.g., generating inappropriate or harmful content like weapons in a children’s context).

The high-level architecture of AI Wallpapers illustrates this layered approach. User selections from the simple frontend are sent to a prompt engineering service, which translates them into structured, detailed prompts, often injecting artistic modifiers and negative keywords for quality control. These prompts then pass through a backend service with the first layer of fast, server-side guardrails (classifiers, blocklists) for safety checks. Once cleared, the prompt goes to an inference platform, where a fine-tuned media generation service creates the image. This platform also hosts a second safety layer: visual classifiers that analyze the output image for safety before an upscaling step enhances resolution. This dual-layer guardrail strategy, combined with a specialized model for quality, allowed for a safe, easy-to-use product that could even be opened to users aged 13 and above.

Tackling the quality challenge involved recognizing that "models aren’t all made equal." Like artists, they excel at certain styles and struggle with others. The team identified the model’s "sweet spots" – themes and categories where it consistently produced outstanding results (e.g., abstract art, specific animals). User choices were then constrained to these domains, guaranteeing high-quality outputs and simplifying quality control. This was validated through large-scale human-in-the-loop testing, generating thousands of images with various stylistic modifiers and concepts. Human designers and researchers reviewed these images not just for technical correctness but for aesthetic quality and "wow factor," using explicit evaluation criteria. This iterative process of human feedback refining automated tests was essential.

This learning was then translated into prompt keywords and a structured prompt format. The team meticulously generated and reviewed over a million images to determine which few words or chip suggestions consistently yielded beautiful, reliable results. This structure not only empowered users without requiring prompt expertise but also enabled engineering optimizations like prefix caching, improving inference efficiency. Collaboration with internal research teams established a powerful feedback loop, improving core models and allowing for targeted fine-tuning where extra polish was needed.

The entire journey from UX exploration to architectural scaling was built with a global user base in mind. A rigorous evaluation pipeline with human raters defined consistent high-quality results. Crucially, the feature was engineered to use a scalable endpoint reused across multiple Google products, reducing overall cost and improving reliability. The launch was met with incredible user praise, validating the non-obvious bet on partially restricted input, and achieved "Google scale," integrating across various surfaces and form factors, and adopted by major OEMs like Samsung.

Case Study 2: Circle to Search – The Understanding Challenge

Transitioning from generative AI to understanding, Circle to Search represents another phase of AI engineering. Launched in early 2024, this feature allows users to instantly search for anything they see on their mobile screen without leaving their current app. From an engineering perspective, it is a complex, real-time, multimodal retrieval task, translating a visual input (a gesture) on a complex screen into a query that delivers high-fidelity, contextual results.

The motivation stemmed from the inherent friction in traditional visual search workflows: taking a screenshot, switching apps, uploading the image, and then searching. Each step was a potential abandonment point. The project’s North Star was "zero context switching, zero interruption, and a seamless flow from curiosity to answer." This vision was achievable by building upon Google Lens, which already possessed world-class visual understanding, but adding deeper AI integration and system-level OS integration.

Circle to Search is the natural evolution of Google Lens. While Lens is "world-facing" (searching through the camera), Circle to Search is "screen-facing," turning whatever is on the screen into the query. This simple shift, replacing multiple steps with a single gesture, transforms the task into a real-time retrieval problem demanding extremely low latency and highly relevant results.

The high-level architecture begins on-device, where a user gesture (circle, tap) triggers the capture of "multimodal prompt" – not just pixels, but also the structural layout of the device screen. This rich prompt is sent to the AI pipeline, which performs core AI reasoning and heavy lifting. The final answer is then seamlessly returned to the device as an overlay.

Within the AI pipeline, the multimodal input first hits a core multimodal model, fine-tuned to understand the relationship between the image, the user’s implicit or explicit question, and the screen’s context. Once intent is understood, it kicks off parallel reasoning and retrieval across multiple sources: querying the Knowledge Graph for structured facts, and web and image search for visually similar items, products, and articles. For instance, circling sneakers triggers queries to identify brand and model, simultaneously searching for shopping links, reviews, and photos. The final stage is synthesis and delivery, where an AI overview synthesis model crafts a natural language answer. Before delivery, all answers pass through safety and responsibility filters – the runtime guardrails ensuring helpful and harmless output.

Key AI technologies powering this include MUM (Multitask Unified Model), which moves beyond simple keyword matching to intent understanding across modalities (e.g., understanding "blue" as a text query applied to a visual of shoes). The integration of Gemini powers richer AI overviews and contextual help, elevating the tool from simple identification to complex reasoning.

A game-changer for Circle to Search was its deep system-level integration directly into the Android OS. This provided a universal, effortless gesture that works everywhere, eliminating the friction of the old workflow. Furthermore, system-level integration provides the AI with a deeper level of context: when an object is circled, the system knows exactly which specific pixels belong to that object within the semantic layout of the entire screen, leading to highly accurate search results.

Guardrails for Circle to Search focus on reliability and safety. Firstly, the visual AI is grounded in factual data by cross-referencing recognized entities with the structured Knowledge Graph, preventing errors like misidentifying a plant. This factual grounding is a primary defense against misinformation. Secondly, a dedicated safety layer ensures every synthesized result, especially those using generative models, passes through a responsible AI filter before being shown. Finally, the system is designed for ambiguity, gracefully handling low-confidence scenarios by offering broad suggestions rather than providing potentially wrong answers, thereby building user trust.

The hypothesis that removing friction through deep system-level integration would unlock new levels of engagement proved correct. Circle to Search achieved massive scale, launching on over 300 million devices, driving a 70% year-over-year increase in visual searches. It has rapidly become a fast-growing query type, particularly popular with younger users for shopping and translation tasks.

Key Learnings and Actionable Takeaways for the AI Industry

The development of AI Wallpapers and Circle to Search offers distinct yet complementary lessons for the broader AI engineering landscape. From AI Wallpapers, the generative product, the primary lesson was the power of clear product principles and tight definitions in managing the "blank canvas problem." The non-obvious bet on guided creation and an obsessive focus on the evaluation pipeline for artistic quality were critical. Over-investing in moments of delight transformed a restrictive experience into something magical.

From Circle to Search, the understanding product, the main takeaway was the transformative impact of deep OS integration. While model improvements were important, the seamless, frictionless invocation of the feature through system-level integration was what truly resonated with users. This integration also unlocked a deeper level of intelligence by providing semantic layout context of the screen. Building trust was achieved by grounding visual AI in factual data, cross-referencing model perceptions with structured knowledge bases to ensure reliability.

These experiences yield actionable takeaways for various roles within the tech industry:

• For AI Engineers: The imperative is to "productize" AI, not just deploy it. A powerful model is merely the starting point; the real work involves building the entire surrounding system, including evaluation pipelines, cost/latency optimizations, and safety guardrails, to deliver a reliable and trustworthy user experience.
• For Tech Leaders: Championing principle-led development is crucial. The difficult decision to adopt guided creation for AI Wallpapers was driven by core principles of quality and safety. Leaders must define these guiding principles and empower their teams to adhere to them, especially when facing challenging choices.
• For Product Managers and Designers: Obsessing over the user’s workflow is paramount. As seen with Circle to Search, identifying and eliminating friction in the user’s journey, enabled by deep integration, was as critical as the underlying AI’s capabilities.

The development process for such complex features is often extensive. AI Wallpapers, for instance, took approximately a year to build, especially given that foundational models were less advanced two-and-a-half years ago. This extended timeline allowed for rigorous quality assurance and continuous iteration based on human feedback, ensuring that outputs met high standards. Testing methodologies involved a continuous evolving loop where human reviewers evaluated pipeline outputs, providing feedback that refined prompt modifiers and model behavior. Cost management, a significant challenge with early generative models, was addressed by migrating to more efficient models, batching prompts, leveraging techniques like prefix caching, and reusing scalable internal endpoints across multiple Google products, thus reducing both operational and development costs. As AI models continue to become more efficient, these costs are expected to decrease further.

In an era where AI is rapidly permeating every aspect of digital life, Google’s approach to engineering AI for mobile—balancing innovation with meticulous attention to user experience, safety, and scalability—sets a significant precedent for the industry. The journey from foundational models to beloved consumer features is complex, but as these case studies demonstrate, a principled, user-centric, and robust engineering methodology is key to unlocking AI’s full potential on mobile.