Your Drupal chatbot has a thumbs up and thumbs down under every answer. Someone clicks dislike. Now what? For most teams the honest answer is: reread the conversation log and guess. The exact prompt that was sent, the documents that were retrieved, and the model and settings active at that moment are already gone by the time anyone goes looking.
This talk starts from first principles. What does observability actually mean, and how is it different from logging, monitoring, and alerting? What is a trace, what is a span, and why do these ideas exist in the first place? Why can an AI feature return a fast, valid response and still be completely wrong, in a way none of your normal dashboards will ever catch?
From there we build the vocabulary specific to AI systems: what needs to be captured about a single model call for it to be debuggable later (the prompt, the retrieved context, the model, provider, and parameters, the token count, the cost, the latency), and why a shared identifier tying all of that back to one user session is what makes it useful after the fact instead of just more noise.
Then we bring it into Drupal. We look at what it actually takes to set up this kind of observability on a real Drupal site: what kinds of tools exist for this today, what a basic setup looks like, and how you would wire something as simple as a dislike click on a chatbot answer back to the exact trace that explains it. The same approach extends naturally to retrieval based search and to multi step AI agents, where a single answer can involve many calls instead of one.
You'll walk away with:
- A working vocabulary: telemetry, traces, spans, and correlation ids, and how these differ from plain logs and metrics.
- The specific things an AI feature needs to record to be debuggable later: prompts, retrieved context, model and provider details, tokens, cost, and latency.
- Why observability and evaluation are two different jobs, and why you need both.
- A practical starting point for setting this up on a real Drupal site, extendable to retrieval based search and multi step agents.
Who this is for
Drupal developers building or maintaining AI powered features (chatbots, agents, retrieval based search, AI assisted content tooling) who've shipped something that works in the demo and want it to be debuggable in production.