Telemetry

About Telemetry

Telemetry is the feature that reads what the Log stores and counts it. It needs no configuration of its own: it reads the Log's destination.

Reading is separate from delivery. Delivery sends events out of a process; Telemetry gathers what was stored and answers questions about it.

The read path has four steps: a source pages through stored lines, a normaliser turns each line into a row (or sets it aside), a cache keeps the rows, and questions read the rows. vinaya sync runs the first three. Every step is pure except the parts that open a folder, call a server or open a database.

How it works

1. 1

   Read a page from the source

   The source is your folder or your server, whichever logs names. Each page returns lines with their positions, a cursor to resume from, and every gap the source knows it cannot fill. Reading never changes the source.

2. 2

   Normalise each line

   Each line becomes one dataset row or one quarantine record, using the same check the sender uses. There is no second validator.

3. 3

   Store it in the cache

   Rows are stored by the event's identity, so reading the same line twice adds nothing. The cache also keeps each source's cursor and every gap.

4. 4

   Ask a question

   A question reads the whole dataset and answers with a figure, or with "unknown" and the reason.

Run a sync

vinaya sync

Each run is bounded: at most 50 pages of 1000 lines, with a look-back of 1000 lines. A run that stops at the page limit says more is available.

The summary counts pages, rows stored, duplicates, edits, deletions, gaps and quarantined lines, and says whether the run completed, whether more is available, and any failure.

What a sync notices

• It re-reads a little. The stored cursor trails what was read by up to the look-back span, so each run re-reads that span.
• It stores before it moves on. Every page is stored first; the cursor moves only after all pages are stored. A failed run keeps its stored pages and the old cursor. A sync never throws for a lost, edited or quarantined line.
• Edits. A known event that comes back with different content is recorded next to the row it kept.
• Deletions. A known event inside the re-read span that the source no longer returns is listed in the summary. Rows are never removed. The span is counted in source positions per stream, never in event times.
• Gaps. Any span the source says it cannot fill, or a source whose head is behind the cursor. A run that saw any gap reports no deletions.

Rows and quarantine

A quarantine record holds the reason, the redacted raw text, the identity and schema version when they can be read, and a sha256 of the raw text. It produces no row.

Rules every row follows:

• Any family works. kind and event are plain text, and nothing on the read path depends on the family. New families and custom events need no code change.
• Identity is the event's, never its position in a file.
• Unknown, never default. A field the line's schema never had is null and listed in the row's unknown map with a reason. Nothing defaults to 0, '' or false. A measured value is either known with a value, or unknown with a reason.
• Low trust. A row written by a CLI older than 0.33.0, or with an unreadable meta.vinaya, has trust: 'low'. Otherwise trust is the header's provenance. A schema 1 row at or above that version has unknown trust.

The cache

Storing a row is idempotent by identity:

Quarantine records are keyed by the hash of their raw text. Gaps are stored once each. The dataset returns rows ordered by time then identity, gaps in the order they were recorded, and quarantined lines in the order they were first stored.

The cache lives in <runtimeDir>/logs-cache/cache.sqlite, never inside the repository or a Log folder.

Questions

A question reads the dataset and selects rows by comparing kind and event as text. A unit of work is workRef. Answers describe what happened; they never claim a cause.

Every answer says how much it rests on: rows read, low-trust rows left out, rows with no known unit, rows used, gaps, quarantined lines and unknowns.

Every question is tested end to end over sample lines, a real folder and a stand-in server, synced into real SQLite caches, including rebuilds.

Configuration

What's next