Data sources architecture

Every telemetry source Landfall reads from, bundled or hosted by you, is one typed contract: a set of advertised operations, each declaring what kind of data it reads, a canonical way to list and range-query each kind, and a raw payload that always reaches the investigator verbatim. This page describes that contract, the catalog built from it, the wire protocol that lets a source live anywhere, and the marketplace an organization chooses its sources from. The setup guides link back here for the shared parts.

Why a typed contract, not just a connector

The connector framework answers how a provider is connected and who may read it. It says nothing about what an operation reads. Without that, every consumer that needs to know "is this metrics or logs, and how do I ask" re-derives it per vendor: discovery tools speak one vendor's vocabulary, several places parse one vendor's response shape, and a capability wall carries a frozen list. Declaring the kind on the source, once, is what lets a Datadog, a Coralogix and a service you wrote last week be discovered and charted by the same code.

Kinds

Every operation a source advertises declares exactly one kind from a closed set:

A source may declare several kinds. A source that declares metrics or logs must provide that kind's canonical operations, and must declare, on itself, how it names and encodes its time-range parameters (the window declaration). The platform applies a viewer-chosen window from that declaration and never consults a central per-vendor table.

The catalog every agent reads

The catalog is the list of an organization's connected sources, scoped to an incident. Each entry carries the source, the connection (and member account where relevant), a label, an optional trusted hint, the source's kinds, and its operations with their kinds. It is built fresh for every investigation from what is actually connected, so a new connection appears in it with no code change anywhere.

Three readers see the same catalog, byte for byte: Beacon, the platform's own investigator; an agent that joins the war room from a terminal through Landfall Edge (and the landfall CLI); and an agent that joins through the hosted manager connector. The joining agents read it through get_signal_catalog and query it through query_signals, both of which reuse the existing gated read routes under the joining agent's grant. A read is refused with the same reasons the routes give: no grant, resolved incident, source not connected, operation not advertised, source not installed.

Canonical operations and native operations

A source's native operations are its own vocabulary: getMetricStatistics on CloudWatch, rangeQuery on Prometheus, queryRange on Loki. They are what the catalog advertises and what an agent's free-form query names. Nothing about them changes, and no source has to alias another vendor's names to be found.

The canonical operations are the per-kind calls a source provides on top: for metrics, list and range; for logs, streams and range. They exist so a generic consumer (discovery, a chart widget, the incident watcher, the live-metrics poller) can ask every source the same question without knowing its vocabulary. Each canonical call delegates to the source's own native operation, so audit, scope and provenance are identical either way, and the catalog marks which native operation backs each canonical one so prompts and generated widget code can prefer it.

Raw on the wire, shaping owned by the source

Every read returns the provider's raw response verbatim in the standard result envelope (source, operation, params, fetched-at, raw, and a partial or error state). That is a resilience property: a vendor changing its response format cannot break the platform, because the platform never depended on the format.

A source may additionally offer a shape adapter: a best-effort function that turns its raw metrics response into a common series shape (label, unit, points of time and value) or its raw logs response into a common entry shape (time, line, labels, stream). Adapters are optional and owned by the source. A consumer that gets nothing back, or an exception, falls back to the raw payload rather than failing the read. Widgets, the watcher and the live-metrics poller obtain series and entries only this way; the platform contains no per-vendor response parsing outside the source packages, and a guard test fails if any is reintroduced.

How a source registers

Three paths lead to the same catalog entry:

1. A core package. A source Landfall ships, as one package folder in the signals library: its plugin (operations with kinds, window declaration, capability objects, shape adapters), its connection adapter (config schema and verify) and its docs guide. Adding one touches that folder and its guide, and no platform file. The fake source used by the local harness is registered this way too, behind a dev flag.
2. A marketplace package, later. The same package shape, built by a screened third party, submitted, built in isolation, conformance- and security-screened, signed, and indexed into a release image. The data model leaves room for it; the path itself is not built yet.
3. A remote source, by URL. A service you host that implements the Source Protocol. Landfall's one generic remote source reads the service's own capabilities at connection time and advertises exactly those. Nothing is shipped in Landfall for it. See Remote source.

The Source Protocol

The protocol is the wire twin of the in-process contract. It is defined once, as a schema, and rendered as both an HTTP contract (OpenAPI) and a gRPC contract, which carry identical operations and fields: a capabilities listing, a health check, a raw invoke for any advertised operation, and the four canonical calls. Every call is authenticated with the credential the admin supplied at connection time (a bearer token, or mutual TLS), and every response is the same raw envelope, with an optional shaped series or entries field beside the raw payload. The endpoint table, the reference server and the conformance suite are on the Source SDK reference.

A query, end to end

1. An agent (Beacon, or a joining agent through query_signals) names a source, an operation and parameters, under the incident it is working.
2. The scope guard checks that the source is in the incident's catalog and that the operation is advertised; a joining agent's grant is checked live, and a resolved incident refuses. Denials are audited and appear on the room's timeline.
3. The organization's connection for that source is resolved (a specific account or service when there are several) and the source's install state is checked. The caller never sees the credential.
4. If the operation takes a window, the viewer's window is encoded the way the source itself declared.
5. The source's plugin invokes the provider (a vendor API for a bundled source; your service's /v1/invoke or a canonical endpoint for a remote one) within the shared timeout and size budget.
6. The raw response comes back in the envelope with provenance; a failure comes back as a partial envelope with the reason, never a silent empty. The call is audited with the agent as actor and the human as metadata.
7. If a consumer needs a shape (a chart, the watcher), the source's adapter is tried and raw is the fallback.

The marketplace: per-organization enablement

The marketplace lists every source bundled in the running platform, with its name, kinds, transport (bundled, or hosted by the developer), docs link and install state for this organization. An admin installs and uninstalls without a deploy; install state gates the source's wizard card, its trigger door and its catalog entry for that organization only. Uninstall is refused while the organization has an active connection of that source, so no incident loses a source silently. Absence of a record means installed, so every existing organization keeps every source when the marketplace first appears. Members can view; only admins change it, and every change is audited.

The listing surface is designed so a screened third-party package, once that path exists, appears as one more card, with nothing changing in what an admin sees.

Conformance is the contract

One suite runs against any source, bundled or remote: every advertised operation invokes or fails with a typed error; unknown operations fail; windowed operations declare an applicable window; each declared kind's canonical operations are wired; shape adapters return well-ordered results on fixtures; the health check fails closed on empty configuration. It is what makes the interface a contract rather than a convention, and it is what the wizard runs before marking a remote source connected.