|
| 1 | +# Cron Exit Ledger Validation |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +As of this fix, hybrid-memory cron jobs now validate that all required maintenance steps complete successfully before reporting `status: ok`. This prevents the dangerous situation where cron state shows "healthy" while maintenance is actually failing. |
| 6 | + |
| 7 | +## Problem |
| 8 | + |
| 9 | +Previously, cron jobs could report `status: ok` even when: |
| 10 | +- Required maintenance steps failed (non-zero exit codes) |
| 11 | +- Steps were missing from the exit ledger |
| 12 | +- Commands were invalid (`unknown command` errors) |
| 13 | +- Only partial work completed (e.g., only `prune` ran, not `distill`/`extract`/etc.) |
| 14 | +- Guard files weren't updated due to failures |
| 15 | + |
| 16 | +This happened because OpenClaw cron treated "agent produced a response" as success, regardless of what that response contained. |
| 17 | + |
| 18 | +## Solution |
| 19 | + |
| 20 | +### 1. Exit Ledger Validation (`services/cron-exit-validator.ts`) |
| 21 | + |
| 22 | +New module that validates maintenance execution by: |
| 23 | +- Parsing HM_EXIT file for all step results |
| 24 | +- Checking for `unknown command` errors in HM_LOG |
| 25 | +- Validating all required steps are present with `exit=0` |
| 26 | +- Supporting skip variants (e.g., `distill-skipped` when config disabled) |
| 27 | +- Returning structured status: `success`, `partial`, or `failed` |
| 28 | + |
| 29 | +### 2. Updated Cron Message Templates (`services/cron-job-bash-harness.ts`) |
| 30 | + |
| 31 | +Enhanced templates now include: |
| 32 | +- Explicit validation instructions for agents |
| 33 | +- List of required steps that must complete |
| 34 | +- Clear success criteria (all steps present with `exit=0`) |
| 35 | +- Mandatory guard update only after successful validation |
| 36 | +- Structured reply format for debugging |
| 37 | + |
| 38 | +### 3. Message Normalization (`cli/cmd-install.ts`) |
| 39 | + |
| 40 | +When `normalizeExisting=true` (e.g., during `verify --fix`): |
| 41 | +- Updates existing cron job messages to match current definitions |
| 42 | +- Removes obsolete command references (e.g., `consolidate-episodes`) |
| 43 | +- Preserves guard prefixes while updating orchestration instructions |
| 44 | +- Ensures all jobs use latest validation rules |
| 45 | + |
| 46 | +### 4. Validation CLI Command (`cli/commands/manage/register-validate-cron-exit.ts`) |
| 47 | + |
| 48 | +New internal command for validating cron execution: |
| 49 | +```bash |
| 50 | +openclaw hybrid-mem validate-cron-exit \ |
| 51 | + --exit-path ~/.openclaw/logs/cron-hybrid-mem/nightly-memory-sweep-*.exit.txt \ |
| 52 | + --log-path ~/.openclaw/logs/cron-hybrid-mem/nightly-memory-sweep-*.log \ |
| 53 | + --required-steps prune distill extract-daily resolve-contradictions enrich-entities \ |
| 54 | + --allow-skip |
| 55 | +``` |
| 56 | + |
| 57 | +Returns exit code 1 if validation fails. |
| 58 | + |
| 59 | +## Expected Behavior |
| 60 | + |
| 61 | +### Success |
| 62 | + |
| 63 | +All required steps present with `exit=0`: |
| 64 | +``` |
| 65 | +2024-05-08T02:00:00Z prune exit=0 |
| 66 | +2024-05-08T02:01:00Z distill exit=0 |
| 67 | +2024-05-08T02:02:00Z extract-daily exit=0 |
| 68 | +``` |
| 69 | +- `maintenanceStatus`: `success` |
| 70 | +- `guardUpdated`: `true` |
| 71 | +- Cron status: `ok` |
| 72 | + |
| 73 | +### Failed |
| 74 | + |
| 75 | +Any step missing or non-zero exit: |
| 76 | +``` |
| 77 | +2024-05-08T02:00:00Z prune exit=0 |
| 78 | +2024-05-08T02:01:00Z distill exit=1 |
| 79 | +``` |
| 80 | +- `maintenanceStatus`: `failed` |
| 81 | +- `guardUpdated`: `false` |
| 82 | +- Cron status should be: `error` (agent must signal failure) |
| 83 | + |
| 84 | +### All required steps missing (empty or incomplete ledger) |
| 85 | + |
| 86 | +When the exit file exists but **none** of the required steps appear in the ledger (including an empty file), validation returns **`failed`**. That pattern matches an abort before the first `hm_step` wrote to `HM_EXIT`, and must not be treated as a successful skip. |
| 87 | + |
| 88 | +Use config skip variants (e.g. `distill-skipped` with `--allow-skip`) when a step is intentionally omitted. |
| 89 | + |
| 90 | +### Partial |
| 91 | + |
| 92 | +Some steps missing: |
| 93 | +``` |
| 94 | +2024-05-08T02:00:00Z prune exit=0 |
| 95 | +(distill and others missing) |
| 96 | +``` |
| 97 | +- `maintenanceStatus`: `partial` |
| 98 | +- `guardUpdated`: `false` |
| 99 | +- Cron status should be: `error` |
| 100 | + |
| 101 | +## Config-Based Skipping |
| 102 | + |
| 103 | +When a feature is disabled in config, the cron message instructs the agent to replace that step with a skip variant: |
| 104 | + |
| 105 | +```bash |
| 106 | +# If distill.enabled is false: |
| 107 | +hm_step "distill-skipped" bash -c 'echo distill disabled; exit 0' |
| 108 | +``` |
| 109 | + |
| 110 | +With `allowSkip=true`, the validator accepts these skip variants as satisfying the required step. |
| 111 | + |
| 112 | +## Migration Path |
| 113 | + |
| 114 | +Existing installations with old cron job definitions will be updated automatically when: |
| 115 | + |
| 116 | +1. User runs `openclaw hybrid-mem verify --fix` |
| 117 | +2. Plugin upgrade triggers `ensureMaintenanceCronJobs(..., { normalizeExisting: true })` |
| 118 | +3. Gateway restart (sync from persistent guard files) |
| 119 | + |
| 120 | +The message normalization preserves the guard prefix while updating the orchestration body to include validation rules. |
| 121 | + |
| 122 | +## Testing |
| 123 | + |
| 124 | +Comprehensive tests in `tests/cron-exit-validator.test.ts`: |
| 125 | +- Exit line parsing (valid, invalid, non-zero codes) |
| 126 | +- Unknown command detection |
| 127 | +- Success/failed/partial status determination (all-required-missing → failed) |
| 128 | +- Skip variant handling |
| 129 | +- Missing file handling |
| 130 | + |
| 131 | +All 14 tests pass. |
| 132 | + |
| 133 | +## Acceptance Criteria Met |
| 134 | + |
| 135 | +✅ Required-step failure makes cron run non-OK |
| 136 | +✅ Unknown command makes cron run non-OK |
| 137 | +✅ Missing required steps treated as failure |
| 138 | +✅ Empty or incomplete ledgers do not produce a false `ok` / skip outcome |
| 139 | +✅ Message normalization removes obsolete commands |
| 140 | +✅ Validation instructions embedded in all cron messages |
| 141 | + |
| 142 | +## Future Work |
| 143 | + |
| 144 | +To fully close the loop, OpenClaw cron core would need to: |
| 145 | +1. Accept structured status from agent (not just text response) |
| 146 | +2. Map `maintenanceStatus` → cron `status` (`success`→`ok`, `failed`/`partial`→`error`) |
| 147 | +3. Update `lastRunStatus` and `consecutiveErrors` based on maintenance outcome |
| 148 | + |
| 149 | +Until then, agents must signal failure explicitly (e.g., by mentioning "FAILED" in response or exiting with error) when validation fails. |
0 commit comments