fix(vaapi): consider rate control when selecting encoding entrypoint - #5763

Description

On an Intel Skylake HD 520 host, the driver advertises H.264 low-power encoding with CQP only, while normal encoding supports bitrate-controlled modes. Sunshine selects the low-power entrypoint before checking rate-control capabilities, so lowering the client bitrate does not reliably constrain encoded output. The observed symptoms were delayed initial video, repeated unrecoverable frames, and prolonged freezes.

Select an entrypoint that supports the requested rate-control mode before falling back to the existing preference order. In automatic mode, prefer CBR/VBR support for the existing Intel/AV1 whitelist, and CBR elsewhere; retain low-power preference when it offers either. Explicit CQP remains supported, and devices with only low-power encoding retain their existing fallback.

This is a narrower upstream candidate than the locally tested workaround that unconditionally swapped the two entrypoints. It does not change deployed software on the test host.

Related work

Searched open and closed PRs/issues for VAAPI, CQP, entrypoint, and low-power encoding. #5388 adds rate-control configuration after entrypoint selection, but does not address this selection problem. No direct existing fix was found. This is not claimed to be a regression introduced by the latest release: the previous tested release has the same entrypoint preference.

Evidence

Host: Surface Pro 4, Intel HD 520, Arch Linux/Hyprland; intel-media-driver 26.2.4, libva 2.24.1. Original Sunshine: v2026.914.233613.

• FFmpeg H.264 CBR probe: normal entrypoint succeeds; low-power entrypoint reports supported modes: CQP.
• Direct capability probe with this PR's selector: LP entrypoint 8 reports mask 16; normal entrypoint 6 reports mask 3294; selector chooses 6.
• Earlier live sessions had 51–74% client-reported network frame loss. With the two-line normal-entrypoint workaround, the user confirmed smooth operation; a roughly three-minute 1440p60 session with the client set to 80 Mbps reported 0.00% network frame loss, 0.05% jitter drops, 57.20 FPS average, and normal encoding with VBR in host logs. The client bitrate setting is not a measurement of actual wire bitrate.
• The capability-aware PR implementation has unit and live capability-probe coverage, but has not yet had a full end-to-end streaming run or the complete project test suite. Other GPUs have not been tested.

Validation

• Nine GoogleTest cases passed, compiled with C++23, -Wall -Wextra -Werror, and SUNSHINE_BUILD_VAAPI.
• Cases cover LP CQP-only vs normal CBR/VBR, keeping suitable LP, explicit CQP/CBR/VBR, unsupported requested modes, failed/unsupported queries, LP-only hardware, picture fallback, no encoding entrypoints, and preserving automatic CBR-only policy outside the VBR whitelist.
• Candidate vaapi.cpp syntax checked with the existing Linux build's compiler flags and dependencies.
• git diff --check.

Screenshot

Not applicable.

Issues Fixed or Closed

None automatically closed.

Roadmap Issues

None.

Type of Change

• fix: Bug fix
• test: Tests for changed selection behavior

Checklist

• Code follows project formatting
• Self-reviewed
• Doxygen documentation added/updated
• Targeted tests added and run
• Full project test suite and cross-GPU validation

AI Usage

• None
• Light
• Moderate
• Heavy: Codex authored this patch and tests; the user performed the live streaming acceptance tests for the precursor workaround.

This draft follows review in the contributor fork (Alex-wangyang#1) and is submitted upstream at the contributor's explicit request.