Framer AI AgentsDesign and publish professional sites with AI
TraceLLM
OpenTelemetry for production AI applications
149 followers
OpenTelemetry for production AI applications
149 followers
Tracellm is an observability platform for production AI applications. Monitor prompt execution, token consumption, latency, spans, errors, and model calls across your LLM workflows. Export traces using OpenTelemetry (OTLP) and quickly identify bottlenecks before they impact users
TraceLLM
Every question on that list is about a failure that announces itself. Failed request, slow call, token spike, all visible.
The one that costs most in production is the request that succeeded. No error, normal latency, sensible token count, and the answer was confidently wrong. To catch that afterwards the trace has to hold what the turn was grounded in, which retrieval actually landed and what the tool returned, attributed to that turn. Otherwise the worst incident of the year looks like a healthy span.
Does a span carry the retrieved content and tool results, or the call and the timing only?
TraceLLM
@jernej_jan_kocica yes, it can carry the grounding context and tool results, not just call timing, but we make that explicit and configurable because production AI traces can contain sensitive customer data.
@jyotishmoy_deka That tension is real, and it is worse in support than most places, since the retrieved text is often the customer's own data.
Worth splitting though, because the check does not need the content. To tell whether the grounding changed between two turns you only need identity and presence: chunk or document ids, the source, the score, the count, maybe a hash. That answers whether anything was retrieved and whether it was the same thing as last turn, with no customer text stored at all. The content itself can stay opt in for the teams that want it.
If identifiers are always on and the text is the configurable part, you get the cheap check by default and the expensive detail only where someone accepted the risk. Is that roughly the split you have, or is the whole grounding block one toggle?
TraceLLM
@jernej_jan_kocica Today the pieces are there, but the split is not first-class yet.
You can already send retrieval identity/presence data as span metadata, for example document IDs, chunk IDs, source, score, count, or hashes. And you can keep retrieved text in input/output fields so it is controlled separately by content-capture settings.
But your framing is better than a single grounding toggle. The right product shape is: grounding metadata by default, retrieved content opt-in. That gives teams the cheap check without storing customer text.
I'm going to make that split explicit in the SDK/docs/UI rather than treating the whole grounding block as one thing.
@jyotishmoy_deka that is the right split, glad it is going in.
one thing i would put in the default, since it is the whole point of the cheap check: emit the metadata even when nothing came back. a zero count is the signal. if the span only exists when a lookup happened, then no retrieval and no instrumentation look identical in the trace, and the bad case is exactly the one where no lookup ran.
thanks for taking it seriously, i will watch for it in the sdk.
TraceLLM
OTLP as the export path is the right bet, but it moves the hard problem rather than solving it. The moment a prompt body leaves for someone's existing stack it lands under retention and access rules it was never collected under, so the first thing I'd want to know is whether payload capture is opt in per span and whether I can strip it at the collector rather than in the SDK. Worth answering on the page, because that's the question that decides whether this gets past a security review or dies in one.
TraceLLM
@asadmalik901 Yes, that's the right boundary.
TraceLLM is privacy-first: prompt/output payload capture is opt-in. By default, spans can carry timing, status, kind, IDs, token usage, and metadata without storing the actual prompt/body/output.
For export, the goal is that payload filtering happens before OTLP leaves TraceLLM. So a project can keep content visible in TraceLLM, or strip content from external OTLP exports, depending on policy. That is safer than depending only on SDK discipline.
Today we support configurable content capture and export policy controls. The part I want to make more explicit is per-span/per-kind payload policy, for example: allow metadata for retrieval/tool spans, but block raw prompt/tool result bodies unless enabled.
I agree this needs to be visible on the page/docs because it is exactly what security teams will ask before adopting this.
@jyotishmoy_deka The part that decides adoption is who owns the switch. If an SDK flag can turn content capture on, one engineer's staging config becomes a prod leak and nobody notices for a quarter. Make the policy server side so the collector drops payloads regardless of what the SDK sent, and show a counter for how many it dropped. Security teams will accept a number they can watch, they won't accept a setting they have to trust.
TraceLLM
@asadmalik901 I totally agree with you. Thank you so much for the suggestions!
The redaction point further up is the one that decides whether we could even turn this on. Full request and response bodies leaving our infra to a third party is a hard no on anything finance adjacent unless PII gets scrubbed before export, not after. Is that per field configurable or a fixed blanket list?
TraceLLM
@raffay_sajjad basic redaction exists today; per-field configurable redaction is the right next step and should be a first-class security feature, not an afterthought.
@jyotishmoy_deka Good to hear it's the direction. For anything finance-adjacent the field-level part matters more than most categories, a prompt built from someone's transaction history has account numbers and merchant names baked into the string itself, not sitting in a discrete column you can just exclude. Scrubbing after the fact means trusting a regex against free text, which isn't something I'd want to defend in a security review.
TraceLLM
@raffay_sajjad facts. Thank you for the suggestions!
Pazi
This looks super useful — OpenTelemetry-based observability for LLM apps is a great idea, especially catching bottlenecks early. Congrats on shipping!
TraceLLM