Unit 2: writer process and SQLite hot store - #8

Length-prefixed JSON. The decoder checks the declared length against the
header before it holds a body, so a four-byte write from any process that
can open the socket cannot ask it to buffer gigabytes. Samples are rebuilt
field by field rather than validated in place, so a frame carrying extra
keys cannot smuggle them into the hot store's insert.

Flushes on whichever comes first, the interval or the batch threshold, with
the interval measured from the oldest waiting sample so no sample waits
longer than it.

The ceiling is a byte budget, not an element count. sqhp's 100,000-line cap
was a byte budget in disguise -- ~80-byte ILP lines, so ~8 MB -- and a
structured sample here is several times larger than a line, so carrying that
number across as elements would carry several times the memory it bounded.
Bytes are measured by serialising rather than estimated from field lengths:
JSON expands a control character to six bytes, so a raw-byte estimate reports
a fraction of the truth for a value full of them, and an estimate below the
truth makes the ceiling fictional in the case it exists for.

The writer skips a batch whose sequence number it already committed, which is
what makes a resend after a lost acknowledgement idempotent. That comparison
only means anything within one run of the plugin: its counter restarts at 1
when it does. A writer outliving its plugin would otherwise compare the new
run's numbers against the old run's, discard every batch as a duplicate,
acknowledge them all, and record nothing while the plugin reported healthy.

The alternative -- a unique index on (ts, context, path, source) -- would also
discard genuinely distinct sub-millisecond samples for one path and source,
which is the loss sqhp's nanosecond counter exists to prevent.

node:sqlite rather than a native dependency: the API is identical on Node 22,
24 and 26 -- what the package supports, what the Signal K container runs, and
what development uses -- verified on all three rather than assumed. That keeps
a compiled module out of the deploy's platform staging.

One transaction per flush, WAL, synchronous=NORMAL: signalk-parquet commits
each record on its own at FULL and pays 35 KB of writes for a 337-byte row.
The sequence number moves inside the same transaction, so a batch that rolls
back is not remembered as applied and its retry is not then discarded as a
duplicate.

No index on the sample table, deliberately. Nothing reads it by key -- the
roll scans a time range and truncates, and sqlite_scanner scans regardless --
so an index would buy nothing and cost the write budget the design is judged
on.

SQLite will not keep a second writer out: two processes can both open the
same file and both write to it, taking per-transaction locks rather than
per-handle ones, so a second writer would interleave its rows and its
sequence numbers with the first's. PRAGMA locking_mode = EXCLUSIVE would stop
that and is unavailable here -- it also blocks readers, and the roll reading
the store while the writer holds it is why the store is SQLite at all. So the
claim is a separate lock file, reclaimed when its holder is gone and refused
loudly when it is alive.

The socket sits in a 0700 directory and is itself 0600. The directory is what
actually gates access, since reaching a socket needs execute permission on the
directory holding it. Node exposes no way to read peer credentials without a
native addon, which this package exists to avoid, so filesystem permission is
the enforcement rather than SO_PEERCRED.

Logs carry counts, path names and error classes and never a recorded value;
the test for that was mutation-checked, since a log-scanning assertion passes
trivially when the failure path it depends on never runs.

Follows the sibling's ilp-writer, including the rule that cost it the most to
learn: close is the single source of truth for a connection ending, because
Node emits it after a failed connect and after a live socket drops, so a
connect-failure path that also reschedules double-counts flaps and runs two
timers. Accepting a connection does not reset the backoff either -- only
surviving stableConnectionMs does.

A batch on the wire when the connection ends is settled against the writer's
last committed sequence number: at or below it the batch landed and its
acknowledgement was lost, so resending would duplicate rows; above it the
writer never had it, so dropping would lose them. Testing that needs the
acknowledgement genuinely lost, so the test puts a proxy in between that
forwards the welcome, lets the batch commit, then swallows the ack and kills
the connection. Mutation-checked: relaxing the comparison fails the test.

The cardinality cap is enforced here, alongside the filter and the rate cap,
because partition count is bounded by not recording rather than by rolling:
the roll holds one Parquet writer per partition and that, more than data
volume, sets its memory peak.

Over the cap a value is dropped rather than the plugin refusing to record.
Refusing would turn one misbehaving source into a total history outage, which
is worse than a bounded subset; dropping silently would be worse still, so
the count reaches the plugin status and the first few names reach the log.

Rows carry the server's receive time, not the delta's own: a boat is a set of
independent clocks, and per-source timestamps make commits land out of order.
Identity rows skip the filter and the rate cap, since an include-mode filter
would silently disable vessel names, but they are still bound by the context
cap -- otherwise an AIS flood walks past the bound entirely.

The plugin now filters, rate-caps and buffers on the Signal K side and hands
batches to a writer process over the socket. The writer is spawned with an
argument array and never through a shell, since a data directory typed into
the Admin UI would otherwise be a command line, and with
--disable-warning=ExperimentalWarning rather than --no-warnings so node:sqlite
stays quiet without hiding real warnings.

Two failures that would otherwise be invisible now report. An unexpected
writer exit sets a plugin error instead of only logging -- without it the
Admin UI keeps showing whatever the status line last said, which is
'Recording'. And the status line refreshes when the connection is established
rather than on its ten-second tick, so enabling the plugin no longer shows
'writer unreachable' for ten seconds over a connection that took milliseconds.

The import guard now also forbids node:sqlite on the plugin's side: it is a
native module, and importing writer/main.js for a constant would put a
database handle and its WAL in the server's heap. writer/contract.js holds
what both sides need.

Found on the device: the client's first connect routinely beats the writer's
listen by a few hundred milliseconds, so every plugin start logged 'flap 1'
and then buffered for two seconds before recording.

Neither half was right. A connect that never succeeded is not a flap --
nothing was established to drop -- and a second of backoff before retrying a
child process the plugin has just spawned is a second of every start spent
waiting for no reason. The initial delay is now 250ms and doubles from there,
so a writer that really is gone still reaches the maximum quickly, and the
log line names the fault it actually saw. Both cases still count towards the
unhealthy report: a writer that never appears is the case most worth
surfacing.

Found by deploying to a device. The lock file held pid 31, written by the
writer in the previous container's PID namespace. The container restarted,
the writer died with it, the lock survived on the bind mount, and in the new
namespace pid 31 was an unrelated live process -- so the liveness check said
'still held' and every subsequent writer refused to start. The plugin
buffered and reported an unreachable writer, permanently. Inside a container
low pids are not occasionally reused; they are reused on every restart.

The socket is the claim now, and connecting to it is the liveness test.
Nothing answers on a dead writer's socket, in any namespace. The remaining
race is two writers starting within milliseconds of each other, which the
plugin does not produce: it spawns exactly one writer per start and Signal K
serialises starts.

The pid file stays, for whoever is reading the device, and nothing decides
anything from it. Its test kills a real listener rather than closing one,
because Node unlinks the socket on a graceful close and there would be
nothing stale to take over.

Measured on a device. With the default recordOthers off, 16 contexts existed
in the hot store holding nothing but a single identity row each -- AIS
vessels, meteo stations, navigation aids. Every one of those becomes a
partition in the roll for one or two rows, which is the small-file cost the
partitioning decision in Unit 3a has to weigh, arriving through a path that
ignores the setting the operator chose.

Identity still skips the path filter and the rate cap, for the reasons it
always did: an include-mode filter would silently disable vessel names, and
the shared throttle would drop most of the opening burst when many targets
appear at once. It no longer skips the record-self and record-others gates.
This is a deliberate divergence from the sibling, which records names
regardless; recorder.ts is ours rather than a copy.

The rule is about not leaking our local infrastructure -- hostnames, domains,
addresses, internal URLs -- not about the marine domain. Vessel names, MMSIs,
AIS targets and navigational-aid names are ordinary test data here, and a
reviewer told to flag them would object to most of the fixtures forever.

Also drops reviews.tools.github-checks.timeout_ms. It does nothing: the
published schema gives github-checks a single property, enabled, and the
string 'timeout' appears nowhere in it. additionalProperties is false only at
the document root, so the key validated clean and was silently discarded --
and the comment beside it, and the commit message that added it, both claimed
a behaviour that never happened.

Three faults on one path, all reachable without an attacker.

The store's session and sequence counter are process-global while the server
tracked session per connection, so a second connection could re-point them.
Reproduced: one hello and one batch at seq 9007199254740000 from a second
connection, and every later legitimate batch came back {"stored":0} while the
plugin reported "Recording" -- exactly what the handshake was added to
prevent, reached from a socket rather than from a stale writer. The design
says there is one client, so it is now enforced: a new session closes the
incumbent, and a second hello on an established connection is refused.

beginSession runs SELECT, INSERT and UPDATE and was unguarded, unlike the
batch path. A throw there escapes the socket's data handler, is not covered by
main()'s catch, and kills the writer, which nothing respawns.

Failures reported err.name, which is the constant "Error" for every
node:sqlite fault -- a full disk and a corrupt store read identically.
Measured: the cause is in err.code and err.errstr, neither of which can carry
a recorded value.

Four faults, all of which left the status line saying "Recording".

send()'s catch claimed to requeue the batch and tell the plugin, and did
neither -- pump() had already spliced those samples out, so they vanished with
no drop count. Requeueing would retry the same refusal forever, since
encodeFrame is deterministic, so they are discarded, counted through the same
figure the status line reads, and reported.

The buffer admitted samples no frame could carry: maxBytes defaults to 8 MB
and MAX_FRAME_BYTES is 4 MiB. Admission now takes the lower of the two, and
take() bounds a batch by bytes as well as by count, since batchSize is
operator-editable with no ceiling.

markUnhealthy latched on a boolean cleared only by the stability timer, which
is armed on connect and fires once. A fault later in a long-lived connection
therefore latched the plugin red for the rest of that connection and
suppressed every later, different fault. Recovery is now an acknowledgement,
and a changed reason is reported rather than swallowed.

Nothing expired an unacknowledged batch, so a writer that was connected but
had stopped reading -- blocked in a long synchronous SQLite call, or stalled
on the card -- held the socket open forever: close never fired, the batch was
never settled, no flap was counted.

Marine devices have no RTC: fake-hwclock restores a saved time at boot and
gpsd or NTP then steps it, in either direction. Throttle read a backwards step
as a sample arriving too soon, and the drop path returns before recording the
pair, so the path stayed dropped until wall-clock time caught up.

Measured against a one-hour step: every recorded path stopped for the full
hour, no log line, the plugin still reporting "Recording". Now admitted, and
the pair is re-anchored to the new clock so throttling resumes rather than
admitting everything from there on.

This diverges from the sibling copy, which has the same defect; it is tracked
alongside the other two divergences in AGENTS.md.

AGENTS.md still described the pid lock file that was replaced by the socket
claim two commits later, and the README Status section still said the build
records nothing while the same file's opening paragraph described the writer.
Both were written before the code they describe and not revisited.

engines.node was >=22, but node:sqlite needs --experimental-sqlite below
22.13.0 and the writer imports it unflagged. I verified the API on 22.23, 24.19
and 26.7 and concluded it was identical across the range -- which tested the
versions in play rather than the range declared.

admitContext ran ahead of the path filter, so a vessel whose every path was
excluded still spent a cardinality slot and was counted as being recorded. The
cap bounds partition count, and a context that produces no row produces no
partition -- the same reasoning the identity branch already applies one gate
earlier. Both caps now sit behind the filter.

The count-shaped options were validated as 'positive and finite', so 0.5
survived. splice() applies ToIntegerOrInfinity to its delete count, so a batch
size of 0.5 removed nothing while isDue still reported a full batch: the buffer
never drained, grew to its ceiling and started evicting, with the plugin
reporting 'Recording' throughout. Milliseconds and megabytes stay fractional,
where it is harmless.

A ROLLBACK inside the catch could throw and replace the error that caused it.
SQLITE_FULL and SQLITE_IOERR can roll the transaction back themselves, after
which the explicit rollback fails with 'no transaction is active' -- so a
disk-full was reported as a bookkeeping problem.

The data directories were created without a mode, so they landed at 0755 and
hot.sqlite at 0644, and only the socket's parent was ever chmod'ed. The store
was therefore world-readable for as long as the writer took to boot -- a
descriptor opened in that window survives the chmod -- and the directory the
Parquet tree will live in stayed readable permanently. The whole layout is now
created at 0700 and chmod'ed, since mkdir's mode is masked by umask and does
nothing for a directory that already exists.

probeLiveWriter read every connect error as 'no writer here', so EMFILE on the
writer, EAGAIN against a saturated backlog or EACCES would unlink a live
writer's socket and bind a second one over it -- both processes then holding
the store, which is what the claim exists to prevent, with the loser being the
running writer. Only ECONNREFUSED and ENOENT now mean dead, and a probe that
neither connects nor errors resolves as live after a deadline rather than
leaving listen unresolved forever.

Protocol errors interpolated peer-controlled text unbounded into messages that
reach the Signal K log: one frame with a 200,000-character kind produced a
~200 KB line, and V8's JSON parse error embeds raw body bytes including
newlines, so a peer could forge a line. Echoes are truncated and stripped of
control characters, and the parse failure reports its class and the body size
instead of V8's message.

stop() discarded the buffer and any in-flight batch, then sent SIGTERM and
returned without waiting. Three consequences, all on the path Signal K takes
for every config save.

The buffer is sent now, bounded by a deadline so a writer that has stopped
answering cannot hold the server's shutdown open. Discarding it lost several
hundred samples per save at the measured rate, while the configuration
describes the flush interval as the window a power cut loses.

The writer's exit is awaited, with a SIGKILL escalation. Not waiting left the
next start's probe finding the outgoing writer still listening.

EXIT_LOCKED is retried with backoff before being called fatal. It is usually a
predecessor still exiting rather than a real second writer, and latching it
stopped recording for a condition that resolves itself in milliseconds -- with
the operator's only remedy, toggling the plugin, re-running the race.

A welcome can no longer raise the sequence counter above what this run has
sent. A higher value cannot be true, and trusting it counted never-written
samples as stored and pushed the counter past MAX_SAFE_INTEGER, after which
the writer's own validator rejects every batch and the client reconnect-loops
forever.

The drain is mutation-checked: removing it fails with 'the buffer was
discarded rather than sent'.

sample() was copy-pasted into five suites and every copy ended in 'as Sample',
which switches the discriminated union off: sample({kind: "string", value:
4.2}) compiled in all five, which is exactly the mismatch value_kind exists to
make impossible. The copies had already drifted -- three used
context: "vessels.self" against two using "self", so three described a sample
the recorder never produces.

The shared builder is overloaded per kind rather than cast, so a wrong pairing
is a build error again; verified by compiling two deliberate mismatches, which
now fail with 'No overload matches this call'. The remaining casts in
hot-store.test.ts are deliberate -- they construct a kind the schema's CHECK
must refuse.

The fact that the copied modules differ from the sibling is current state and
belongs next to the modules it describes -- a maintainer editing them needs it
before assuming the copies match. The 'fold them back when the issue lands'
half was a to-do, and that does belong in the issue.