Release promotion
Compare a candidate plugin version against stable on eval pass rate, errors, latency, outcome state, and feedback.
Telemetry
Plugin Telemetry for AI Agents
Plugin telemetry tells owners whether an agent capability is being used, where it fails, how long it takes, and whether a new release improved the work.
Publish the product
Use Telvine when the plugin is ready to be treated like software.
The first useful milestone is simple: one plugin directory, one owner, one release record, and one measurement model.
npm i -g @telvine/cli telvine login telvine publish ./my-plugin
What to measure
Useful telemetry starts at observable boundaries: a Skill invocation, a connector call, an MCP server action, a hook run, a command execution, or a package validation step. Each event should describe what happened without copying the work product into analytics.
Telvine uses a metadata-only model so teams can measure adoption and quality without sending prompts, file contents, connector payloads, tool arguments, or model outputs.
The event shape
Skill behavior should use skill.* events. Non-Skill component behavior should use plugin.component.invoked and plugin.component.error. That keeps the app-level product and the execution-level component model clear.
The goal is not a giant event stream. The goal is enough trustworthy evidence to compare versions, spot regressions, and decide what to improve next.
- Plugin id, version, component id, and component type.
- Invocation status, duration, error category, and retry signal.
- Outcome state, feedback score, eval suite id, and promotion gate result.
- No prompts, no source files, no connector payloads, no tool arguments, no model output.
How telemetry changes releases
Without plugin telemetry, teams judge releases by demos and anecdotes. With plugin telemetry, owners can compare candidate and stable versions on actual behavior.
That is the difference between shipping agent capability and operating it.
Where it applies
Use cases that deserve a maintained plugin
Customer success
See which installed plugins are adopted, ignored, failing, or getting better after changes.
Security review
Verify component inventory and event coverage without collecting sensitive work content.
FAQ
Common questions
- What is AI agent telemetry?
- It is metadata about how agent capabilities run: invocations, errors, latency, outcomes, feedback, evals, and version behavior.
- What should plugin telemetry avoid?
- Do not emit prompts, file contents, connector payloads, tool arguments, or model outputs. Use typed metadata instead.
- Can Telvine forward events to existing analytics tools?
- Yes. Telvine can normalize plugin-native events while your existing analytics stack remains the broader system of record.
Related pages