Monitoring & incidents · wired today

What can agents actually do with LangSmith?

Incidents move faster than humans can tab-switch. The LangSmith connector is one of 205 executable connectors and gives agents 10 read-only tools, with the credential held in the vault and every call receipted.

The LangSmith tool list, exactly as agents see it

list_projects — List LangSmith tracing projects (tracer sessions) in the workspace the API key belongs to, newest activity first
get_project — Get one LangSmith tracing project by its session ID, with optional run count, latency, token and cost stats
list_datasets — List LangSmith evaluation datasets in the workspace, with example and experiment counts
list_examples — List examples (inputs and reference outputs) in one LangSmith dataset
get_run — Get one LangSmith run (trace span) by ID, with status, latency window, error, token counts and cost
list_feedback — List LangSmith feedback records, optionally scoped to one run, one project or one feedback key
list_annotation_queues — List LangSmith annotation queues with their queue type, reviewer settings and pending run totals
workspace_stats — Get counts of datasets, tracing projects, prompts, annotation queues and evaluators for the current LangSmith workspace
search_prompts — Search LangSmith Hub prompt repositories by text, returning handles, owners and download counts
server_info — Get the LangSmith instance version, build SHA and maximum supported SDK versions (public endpoint, no credential required)

Every tool above is read-only. Write surfaces ship only after live testing, and always behind approval gates.

What the LangSmith connector needs

You provide API key from your LangSmith account; Quox sends it as an API key sent as a header. The credential is stored encrypted in the vault and resolved just-in-time, so agents use it without ever seeing it.

Where to get it: LangSmith's own API documentation ↗ covers creating and scoping the credential.

How proven is this?

This connector was blind shape-proven against the real LangSmith API: every endpoint it declares answered measurably differently from a deliberately bogus control request, without any account or credential involved. It is Beta: proven against the live API surface, not yet run with a real credential.

LangSmith integration FAQ

What can Quox agents do with LangSmith?

Agents can run 10 read-only tools against LangSmith: list projects, get project, list datasets, list examples, get run and more. Every call is receipted in the evidence trail.

What does the LangSmith integration need?

You provide API key from your LangSmith account; Quox sends it as an API key sent as a header. The credential is stored encrypted in the vault and resolved just-in-time, so agents use it without ever seeing it.

Do agents see my LangSmith credentials?

No. Credentials live encrypted in the Quox vault and are resolved server-side at call time. Agents invoke tools; they never receive the underlying secret.

How proven is the LangSmith connector?