fix: process Delete event for app already deleted on the agent - #1101

What does this PR do / why we need it:

Problem

When an Application is deleted on the principal and, at about the same time, by a user on the managed agent, the Delete event from the principal reaches the agent after the Application is already gone from the workload cluster. The agent fails to process that event, so it never learns that the deletion was legitimate. The concurrent local deletion is then treated as an unauthorized deletion and the Application is recreated; every further attempt to delete it is reverted the same way. The Application no longer exists on the principal, so nothing can ever remove it again until the agent is restarted.

Root cause

agent/inbound.go:875 (deleteApplication): the lookup of the existing Application via a.appManager.Get returns the NotFound error straight to the caller. Everything that tells the agent the deletion is expected runs only after that lookup succeeds: a.deletions.MarkExpected(...) and a.sourceCache.Application.Delete(...). As a result the source UID of the deleted Application is never marked as an expected deletion and the stale source cache entry is kept, and addAppDeletionToQueue in agent/outbound.go recreates the Application through RevertUserInitiatedDeletion.

The NotFound case is already handled when Delete itself reports NotFound a few lines further down; only the Get path was missing it.

Fix

The Delete event path now goes through a new deleteApplicationFromEvent. It calls deleteApplication and, if that returns NotFound from Get, marks the deletion as expected for the source UID carried by the incoming event (sourceUIDForApp(app), which is the principal side UID and therefore the value of the source-uid annotation on the agent side copy), drops the stale source cache entry in managed mode, unmanages the app, logs at debug level and returns nil. Errors other than NotFound are still returned unchanged.

deleteApplication itself keeps returning NotFound from Get. It is also called on a source UID mismatch, where the incoming app is the new application; marking its source UID as expected there would let a later user-initiated deletion of the recreated app go unreverted (pointed out in review). That path therefore behaves as on main. Per review, the app is now also unmanaged in both NotFound cases.

Which issue(s) this PR fixes:

Fixes #1056

How to test changes / Special notes to the reviewer:

How tested

Added the unit test Delete: App already deleted on the agent must not be recreated to Test_ProcessIncomingAppWithUIDMismatch in agent/inbound_test.go. It seeds the source cache with the incoming UID, makes the backend Get return NotFound, sends a Delete event through processIncomingApplication, and asserts that no error is returned, that no Delete/Create backend call is made, that the source cache entry is gone and that the deletion is marked as expected.

Added Create: Old app gone before delete on UID mismatch must not mark the new UID as expected to the same test: the identity check sees the old app, the second Get returns NotFound, and it asserts the error is surfaced, no Delete/Create is made and neither the new nor the old source UID is marked as an expected deletion.

Before the fix:

--- FAIL: Test_ProcessIncomingAppWithUIDMismatch (0.00s)
    --- FAIL: Test_ProcessIncomingAppWithUIDMismatch/Delete:_App_already_deleted_on_the_agent_must_not_be_recreated (0.00s)
        inbound_test.go:401:
            	Error:      	Received unexpected error:
            	            	application.argoproj.io "test" not found
FAIL	github.com/argoproj-labs/argocd-agent/agent	0.059s

After the fix:

--- PASS: Test_ProcessIncomingAppWithUIDMismatch (0.01s)
    --- PASS: Test_ProcessIncomingAppWithUIDMismatch/Delete:_App_already_deleted_on_the_agent_must_not_be_recreated (0.00s)
ok  	github.com/argoproj-labs/argocd-agent/agent	0.055s

go test ./agent/, go vet ./agent/ and go build ./... pass.

Links

• Race condition when Application is deleted simultaneously on both principal/managed agent: creates undeletable Application on managed-agent #1056

Checklist

• Documentation update is required by this PR (and has been updated) OR no documentation update is required.

This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of application deletion events when the application has already been removed.
  • Prevented unnecessary delete or recreate operations in this scenario.
  • Removed stale application state and correctly recorded the deletion as expected.
  • Improved handling of create events when an existing application disappears during processing, ensuring the operation reports an error without incorrectly recording a deletion.