Filter acceptance tests for installation token capabilities - #14354

Depends on #14320.

Description

GitHub App installation tokens cannot perform operations that require a user account, so running the complete acceptance suite with one currently attempts unsupported user-only behavior. The intention is not for this to be a total solution to pruning test trees, but to allow us to unlock running this in CI in a not totally bonkers way.

This middle layer adds explicit per-script metadata describing whether a user-authenticating token is required. The harness classifies OAuth, classic PAT, fine-grained PAT, and GitHub App user tokens as user-capable, while treating GitHub App installation tokens as installation-only and filtering incompatible scripts. Unsupported token prefixes fail explicitly. As a result, an installation token can run the compatible suite without being granted or attempting user-only behavior.

This PR is the middle layer of a three-layer native stack. It depends on #14320 and intentionally does not add the GitHub Actions workflow.

How did you test this change?

OAuth Token

➜ GH_ACCEPTANCE_HOST=github.com \
  GH_ACCEPTANCE_ORG=gh-acceptance-testing \
  GH_ACCEPTANCE_TOKEN="$(gh auth token --hostname github.com)" \
  go test -tags=acceptance -count=1 ./acceptance
ok      github.com/cli/cli/v2/acceptance        366.276s

Installation Token

Minted using gh token generate and stored in GH_ACCEPTANCE_TOKEN:

➜ GH_ACCEPTANCE_HOST=github.com \
  GH_ACCEPTANCE_ORG=gh-acceptance-testing \
  go test -tags=acceptance -count=1 ./acceptance
ok    github.com/cli/cli/v2/acceptance        325.755s

I exercised the new metadata lint by temporarily adding acceptance/testdata/api/zzz-linter-missing-user-capability.txtar containing only fixture-repo none, without the required capability header. I ran the following command and saw it reject the script (output excerpt; local absolute-path error trace omitted):

$ go test ./acceptance -run '^TestAcceptanceScriptsDeclareUserCapabilityRequirement$' -count=1
--- FAIL: TestAcceptanceScriptsDeclareUserCapabilityRequirement (0.04s)
    --- FAIL: TestAcceptanceScriptsDeclareUserCapabilityRequirement/testdata/api/zzz-linter-missing-user-capability.txtar (0.00s)
        user_capability_test.go:163:
            Error:       Received unexpected error:
                         testdata/api/zzz-linter-missing-user-capability.txtar: first line must be '# requires-user-capability: true' or '# requires-user-capability: false'
            Test:        TestAcceptanceScriptsDeclareUserCapabilityRequirement/testdata/api/zzz-linter-missing-user-capability.txtar
FAIL
FAIL    github.com/cli/cli/v2/acceptance    0.423s
FAIL

I removed the temporary script afterward. This only exercised local metadata validation; it did not run live acceptance tests or access GitHub resources.

Key points

• Every acceptance script must declare its user capability requirement on the first line, making compatibility explicit and mechanically validated.
• Explicitly selecting an incompatible script fails with a targeted message; unfiltered installation-token runs omit incompatible scripts.
• Existing fixture declarations, organization scoping, shared-state safety, clone reductions, and public-repository behavior from Reduce rate limiting and reduce time taken for acceptance tests #14320 are preserved.

Notes for reviewers

Start with acceptance/user_capability_test.go for token classification and metadata validation, then read the filtering path in acceptance/acceptance_test.go. Review #14320 first for the fixture and performance foundation this layer builds on.

I will say that I did see quite some flaky behaviour, but I'm content to merge this and do burn down.

Authorship and follow-up

Who wrote this:

• A human wrote it.
• An agent wrote it under close human direction.
• An agent wrote it independently, and no human has guided the implementation beyond the initial prompt.

Who answers review comments:

• @williammartin will read and reply directly. Name the account.
• An agent will draft replies and @username will read them before they are posted.
• Nobody has explicitly committed to replying.