[9.4] (backport #15423) Give service-runtime components more time to check in before failing an upgrade - #15863

…an upgrade (#15423)

* Give service-runtime components more time to check in before failing an upgrade

Elastic Defend restarting slowly after an upgrade could be mistaken for a
broken Agent build and trigger an unnecessary rollback. The FAILED
threshold for service-runtime components is now derived from their own
configured install/uninstall/check timeouts instead of the generic 90s
window; DEGRADED reporting cadence is unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Cap service check-in failure window to a safe margin below the upgrade watcher grace period

A service-runtime component's failure threshold could still exceed a very
short configured grace period, and the grace period value was captured once
at startup so a live config reload never reached already-running components.
Cap the threshold relative to the grace period and share it live between the
Manager and its components so a reload takes effect immediately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Simplify grace period wiring: drop ManagerOption in favor of a plain setter

The grace period is a mutable, post-construction value now that Reload can
update it live, so threading it through a functional-options mechanism at
construction time added an abstraction the value did not need. A plain
SetServiceCheckinGracePeriod method called once after NewManager achieves
the same result with less API surface.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Declare Reload on the RuntimeManager interface instead of type-asserting

RuntimeManager was the only reloadable subsystem in Coordinator.generateAST
without a Reload method on its own interface, so reloading it required a
type assertion to the concrete *runtime.Manager. Add Reload to the interface
itself, matching UpgradeManager and MonitorManager, and give the test fake a
no-op implementation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Trim comments that leaked implementation detail

Several comments walked through how the grace period sharing/atomics work
instead of just stating what each piece is for. Shortened them to plain,
purpose-focused descriptions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Drop remaining method-name reference from a comment

The field comment still named Reload as the update mechanism, which the
gracePeriodValue type comment already covers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Drop redundant live-update note from a field comment

The gracePeriodValue type comment already explains that the value can be
updated live, so restating it on every field that holds one was redundant.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Add Reload to the generated RuntimeManager mock

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Regenerate RuntimeManager mock via mockery and add it to .mockery.yaml

Adds the coordinator's RuntimeManager interface to .mockery.yaml so it
is regenerated automatically going forward. Runs mockery v3 to produce
the updated mock (includes the new Reload method) and picks up minor
v3-format header updates across other mocks in the same run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Pin service check-in grace period to upgrade marker during active upgrade

Stores the configured watcher grace period in the upgrade marker when an
upgrade starts. The new agent reads it back and uses it as a ceiling on
the check-in miss cap, so a Fleet config reload that increases the grace
period mid-upgrade cannot push component failure detection past the
deadline the watcher already committed to.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Assign upgradeGracePeriod to a variable in NewUpgrader

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Simplify grace period handling: plain field on Manager, skip Reload during upgrade

gracePeriodValue goes back to a simple atomic with no ceiling. Manager
gains a plain upgradeGracePeriod field that is set once from the marker
at startup. Reload skips updating the grace period while an upgrade is
active — the marker value is a constant for this upgrade and live config
changes are irrelevant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Lift upgrade grace period freeze when upgrade reaches terminal state

Elastic Defend can take longer than the hardcoded ~90 s window to restart,
causing the upgrade watcher to roll back a healthy upgrade. This commit
introduces a grace-period freeze so the runtime uses the same window
configured in the upgrade watcher for the duration of an upgrade.

Changes:
- gracePeriodValue on Manager holds both a live value (updated by Reload)
  and an upgrade value (frozen when an upgrade starts, cleared on
  completion). Get() returns the upgrade value while non-zero.
- IsUpgradeActive() on UpdateMarker encapsulates the three conditions
  (not acked, GracePeriod > 0, non-terminal) used at startup to decide
  whether to activate the freeze.
- Coordinator clears the freeze via ClearUpgradeGracePeriod when the
  upgradeMarkerUpdate channel delivers a terminal or nil-Details marker.
- MockRuntimeManager regenerated to include ClearUpgradeGracePeriod.
- Tests: TestManagerGracePeriodUpgradeFreeze, TestUpdateMarkerIsUpgradeActive,
  TestCoordinatorClearUpgradeGracePeriodOnMarkerUpdate.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Remove upgrade marker grace period freeze mechanism

The freeze mechanism (snapshot GracePeriod in the marker, freeze the service
check-in window during active upgrades) was added to prevent a config reload
from shortening the window mid-upgrade. In practice the grace period is never
user-configurable at runtime, so the problem it solved does not exist.

Remove: UpdateMarker.GracePeriod field and serializer counterpart,
IsUpgradeActive(), gracePeriodValue type and its atomic fields, the three
Manager methods (SetServiceCheckinGracePeriod, SetUpgradeGracePeriod,
ClearUpgradeGracePeriod), ClearUpgradeGracePeriod from RuntimeManager
interface, the coordinator handler that called it, and all associated tests.

Simplify: maxCheckinMisses no longer takes a dynamic cap from the manager;
checkinFailureTimeout drops the uninstall operation (only install/check
determine the initial startup window). Bump defaultGracePeriodDuration to
11m so it always exceeds the 10m Elastic Defend operation timeout, and add
TestUpgradeGracePeriodExceedsMaxServiceTimeout to enforce that relationship.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Drop Reload from RuntimeManager; strengthen grace period constraint test

Remove the no-op Reload method from RuntimeManager interface, Manager,
the mock, and fakeRuntimeManager — it was left over from the abandoned
freeze mechanism and never needed.

Make TestUpgradeGracePeriodExceedsMaxServiceTimeout load the real
endpoint-security.spec.yml so a spec change that breaks the grace period
invariant fails the test rather than passing against a hardcoded constant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit eb3502f)