From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

The rapidly evolving landscape of artificial intelligence is challenging developers to rethink their relationship with the tools they create and use. A recent presentation by Ben O’Mahony, a Principal AI Engineer at Thoughtworks, delved into a compelling strategy for personalizing and optimizing AI agent performance: distilling the behavior of large, "frontier" models into smaller, more efficient language models (SLMs) using production telemetry. This approach, detailed at an InfoQ event, promises to revolutionize how organizations build, deploy, and continuously improve their AI systems, addressing critical concerns around cost, latency, privacy, and control.

The Evolving Relationship with AI Tools: Shaping and Being Shaped

O’Mahony commenced his presentation by invoking John Culkin’s paraphrase of Marshall McLuhan: "We shape our tools, and thereafter our tools shape us." This profound observation, decades old, resonates with unprecedented relevance in the era of artificial intelligence. As developers increasingly integrate AI into their workflows, they are not merely adopting new functionalities but fundamentally altering their habits, thought processes, and even their definition of "better work." The critical question, O’Mahony posited, transcends mere functionality: "Does this make you a better person? What kind of person does this tool create?"

Developers, by nature, meticulously craft their environments. O’Mahony shared his personal investment in customizing Neovim with hundreds of lines of Lua and countless plugins, embodying the desire to shape tools to individual preference. Yet, when it came to AI, he encountered a significant limitation: the inability to deeply customize the "brain" of the tool. While models could be swapped for quality improvements, the core AI API remained a black box, resistant to the kind of permanent, iterative refinement that developers apply to traditional software. This frustration, coupled with the escalating token costs of proactive large language model (LLM) interactions, sparked the innovative journey he described.

The AI-Powered Language Server Protocol (LSP): A Practical Testbed

To illustrate his solution, O’Mahony detailed the creation of an AI-powered Language Server Protocol (LSP). The Language Server Protocol, a cornerstone of modern development environments, provides language-specific intelligent features like semantic analysis, inline hints, and intelligent suggestions across various editors such as VS Code, Neovim, and Emacs. Traditional LSPs, like pyright or rust-analyzer, operate on rule-based parsing of abstract syntax trees, offering speed, precision, and reliability for tasks like syntax checking and type inference. However, they inherently lack semantic understanding; they cannot discern whether a function’s name accurately reflects its current operation or identify fragile error handling patterns.

This is where O’Mahony’s AI LSP steps in. Designed for proactive, "agentic" suggestions directly within the developer’s workflow, it eliminates the need for separate chat panels. A key advantage of the AI LSP is its language-agnostic nature, capable of analyzing any code or even Markdown to suggest improvements, including code fixes. O’Mahony demonstrated this through a "meta" example: the AI LSP suggesting improvements to its own codebase, presenting diagnostics and code actions (with multiple options) directly in the editor. This contextual integration prevents flow disruption, offering a superior user experience compared to sifting through lengthy pull requests or disparate CLI outputs.

The AI LSP’s proactive nature, while powerful, also presented challenges. Every file save triggered an interaction with an AI agent, leading to significant token consumption over a full coding day. Moreover, the lack of deep customizability for the underlying AI model limited its ability to truly adapt to O’Mahony’s specific coding style or project idiosyncrasies.

OpenTelemetry: The Foundation of Observability for AI Agents

The breakthrough came with the realization that the very interactions generating these costs were also producing invaluable data. O’Mahony leveraged OpenTelemetry (OTel), the industry standard for observability, to capture every facet of his AI LSP’s operation. OTel provides distributed tracing, allowing developers to follow a request’s journey across an entire system, through multiple services and agents, pinpointing what happened at each step. In traditional web applications, traces might show cache checks or database queries. For generative AI, OTel’s utility is even more profound.

The OTel specification for generative AI explicitly defines how to capture critical data points:

• Events: Inputs and outputs like system messages, user messages, assistant messages, and tool calls.
• Metrics: Token usage, operation duration, request duration, time per output token, and time to first token – all vital for debugging and performance analysis.
• Spans: Model and agent spans that encapsulate operations such as agent creation, workflow execution, chat completion, multi-modal generation, and tool execution.

O’Mahony emphasized that every AI agent should be instrumented with OTel from day one, not just for potential distillation but for immediate debugging and understanding of system behavior. Auto-instrumentation, widely supported by popular AI frameworks and libraries like Logfire and Pydantic AI, simplifies this process, achieving 99% coverage with minimal effort. This instrumentation captures not only technical metrics but, crucially, user actions—the "gold" for training signals.

Building the Data Flywheel: From Telemetry to Training Data

The genius of O’Mahony’s approach lies in transforming routine product usage into a continuous, self-labeling data generation process. By instrumenting specific user interactions within the AI LSP, he established a feedback loop:

• Accepted Fix: When a diagnostic appears and the user clicks "apply fix," it’s a strong positive signal. The user explicitly validates both the problem identification and the proposed solution.
• Regenerated Suggestion: Clicking "regenerate AI suggestion" indicates a weak positive signal. The user agrees with the diagnostic but seeks an alternative solution, implying the AI is on the right track but needs refinement.
• Dismissed Suggestion: This constitutes strong negative feedback. Users explicitly reject the AI’s suggestion, providing invaluable examples of what not to do.

These behavioral feedback signals are far more potent than generic "thumbs up/down" mechanisms, as they reflect actual user behavior within the product workflow. Unlike traditional machine learning, where data labeling is expensive and time-consuming, this "data flywheel" inverts the process. Users, or in O’Mahony’s case, himself, become free, continuous labelers simply by using the tool as intended.

The mechanics involve exporting OTel traces to a compatible backend (Logfire, Honeycomb, Datadog) or locally to disk as JSON. An extraction pipeline then queries full traces where user action spans exist, filtering out ignored diagnostics to focus on accepted, regenerated, and dismissed interactions. The output is a simple JSONL file, with each line representing a training example – no complex preprocessing required, as the necessary information is already embedded within the OTel spans. Within weeks, this process can yield hundreds to thousands of labeled examples.

Model Distillation: Crafting Personalized SLMs

With a sufficiently large dataset, the next step is model distillation. O’Mahony utilized frontier models like Gemini to generate initial suggestions, which were then labeled positively or negatively through his own usage. This labeled dataset then served to fine-tune a smaller language model (SLM), typically in the 7-13 billion parameter range, that already possessed a general understanding of coding. The goal wasn’t to train a model from scratch but to adapt an existing open-source model to his specific codebase, coding style, and eccentricities.

The fine-tuning process, performed locally using tools like MLX, took only a few hours. The key, O’Mahony stressed, is not to over-train. The result is a local, cheaper, and faster SLM. While its quality on complex, novel evaluations might be 80-85% compared to a frontier model, it offers significant advantages:

• Cost Efficiency: Drastically reduces token expenditure.
• Speed: Lower latency for local inference.
• Privacy: Enables local deployment, crucial for client projects with strict data governance or intellectual property concerns.
• Personalization: Learns from specific usage patterns, avoiding suggestions that are consistently dismissed and adapting to individual preferences.

This continuous feedback loop ensures that as the SLM is used, it generates more data, leading to further refinement and increasingly personalized, efficient performance.

Platforming AI Distillation: Scaling the Approach Across Organizations

O’Mahony extended his individual project into a vision for platform capabilities, outlining how this approach could generalize across an enterprise. The core pattern remains consistent:

1. Instrument Agents with OTel: Establish a shared instrumentation layer.
2. Capture User Actions as Training Signals: Design user experiences that naturally generate feedback.
3. Distill Model Behavior into SLMs: Utilize shared extraction pipelines and fine-tuning infrastructure.
4. Deploy with Safety Rails and Iterate: Ensure continuous improvement with appropriate guardrails and evaluation tooling.

This architectural consistency transforms distillation from a one-off effort into an extensible platform utility. New agents can be integrated incrementally by defining user actions, mapping them to a positive/negative feedback scale, and plugging into existing infrastructure. O’Mahony suggested a simple configuration-based approach for engineers to define feedback spans, or even platform-specific SDKs/decorators for standardized instrumentation. The emphasis is on simplifying the user experience for engineers, data scientists, and data engineers, making advanced AI model improvement accessible without requiring deep ML expertise or massive compute budgets.

Economic and Strategic Imperatives

The economic angle of this data flywheel approach is compelling. Traditional machine learning incurs significant costs for labeled data, requiring dedicated labeling processes, infrastructure, and tooling. By contrast, product usage becomes the labeling process itself, with users providing feedback as a natural byproduct. This creates a compounding flywheel: better models drive more usage, which generates more data, leading to even better models. Organizations can shift from linear investment in data collection to building systems that automatically generate and utilize training data, enabling continuous, and potentially autonomous, improvement.

Furthermore, the ability to run models locally addresses growing concerns about data privacy and intellectual property. For enterprises handling sensitive information, sending data to external frontier models is often a non-starter. Distilling models for local deployment offers a secure, compliant alternative, broadening the applicability of AI across industries with stringent regulatory requirements.

The "Year of Agents" and the Future of AI Tooling

Looking ahead, O’Mahony believes 2025 will be "more agents," rather than just "the year of agents." He envisions organizations with dozens of internal AI agents—customer support bots, code review assistants, data analysis tools—each generating telemetry and continuously improving through distillation. Crucially, this platform approach enables agents to learn from each other. The insights gained by a customer support bot about product nuances could inform a documentation generator, with multi-agent workflows creating interconnected learning systems. This fosters a truly "learning organization," where AI tools adapt and evolve based on real-world production usage.

Andrej Karpathy’s assertion that "agents are here to stay" reinforces this vision. While current agents have limitations, the trajectory is clear. The ability to shape these tools, rather than passively be shaped by them, becomes paramount. O’Mahony championed a future of "many different intelligences" rather than a singular Artificial General Intelligence (AGI)—specialized, personalized, portable, and continuously improving SLMs that run locally. This paradigm ensures that as AI becomes central to work, human agency and control over workflows are preserved.

Key Takeaways for Developers and Organizations

O’Mahony concluded with concrete, actionable takeaways:

1. Instrument All Agents with OpenTelemetry: Prioritize OTel instrumentation from day one, capturing everything, especially user actions. Auto-instrumentation provides a significant head start.
2. Design Observable User Experiences: Crucially, integrate feedback mechanisms directly into the product workflow. Explicit actions like "accept," "regenerate," or "dismiss" provide powerful signals, far beyond generic ratings.
3. Embrace Distillation When Data is Sufficient: Model distillation is more accessible and affordable than commonly perceived. The tooling has matured, allowing more engineers to participate. Even a few hundred iterations and a month’s worth of usage data can yield significant improvements for a 7-13 billion parameter model, often achievable locally within hours.

The journey from frontier models to personalized SLMs, powered by production telemetry, represents a paradigm shift. It empowers developers and organizations to move beyond generic AI solutions, crafting intelligent tools that are cheaper, faster, more private, and continuously tailored to their unique needs. This ensures that in the ongoing dance between humanity and technology, we remain firmly in the driving seat, shaping our AI tools even as they, in turn, shape us.