Windows: hevc_qsv encode failure leaves monitor0 stuck permanently, reconnect hangs at "Waiting for image" #16054

Bug Description

I am experiencing a persistent Windows-to-Windows video freeze which appears to be related to the Intel QSV / MFX hardware encoding path.

This has happened on multiple controlled Windows machines using Intel QSV, so it does not appear to be isolated to one PC.

The failure sequence captured in the controlled-side server log is:

1. A desktop session starts normally.
2. RustDesk successfully creates the DXGI capturer and Intel QSV H.265 encoder (hevc_qsv).
3. During the active session, FFmpeg/QSV suddenly reports:

Invalid FrameType:0.

[FFMPEG_VRAM_ENC] avcodec_send_frame failed, ret = Invalid data found when processing input
encode fail: no valid frame, times: 1

4. The current desktop image freezes. FPS and bitrate on the controlling side become 0.
5. The connection may then be closed/reconnected, but the controlled RustDesk process enters a persistent broken state.
6. Reconnecting succeeds at the connection/control level, but remains indefinitely at:

Connected, waiting for image transmission...

7. Other RustDesk services still work. In particular, I can open RustDesk Terminal on the same controlled machine and use it normally while desktop video remains completely unavailable.
8. Restarting only the RustDesk.exe --server child process immediately restores desktop video.

This appears related to #14662, but in this case I was able to capture the exact video encoder failure and the video-service lifecycle before and after the failure.

Important log sequence

Before the failure, the desktop session starts normally:

Enter monitor0 service inner loop
new video service: monitor0
...
Create capturer dxgi|gdi
...
new encoder: VRAM(... vendor: MFX ... data_format: H265 ...)
[FFMPEG_VRAM_ENC] encoder name: hevc_qsv
...
gdi: false
Call snapshot of monitor0 service

Then the encoder fails:

ERROR ... hwcodec ... Invalid FrameType:0.

ERROR ... [FFMPEG_VRAM_ENC] avcodec_send_frame failed, ret = Invalid data found when processing input
ERROR ... video_service.rs ... encode fail: no valid frame, times: 1

When that connection closes, the other services exit:

Connection closed: Peer close
connection loop exited
Input thread exited

Exit remote-printer service inner loop
Exit display service inner loop
Exit clipboard service inner loop

However, there is no corresponding:

Exit monitor0 service inner loop

Immediately reconnecting creates a new connection successfully. Codec negotiation also succeeds and H.265 is selected. RustDesk starts the normal non-video services:

Connection opened
...
usable: vp8=true, av1=true, h264=true, h265=true
...
Start cm
Enter audio service inner loop
Enter display service inner loop
Enter mouse_cursor service inner loop
Enter clipboard service inner loop
Enter remote-printer service inner loop

But on this reconnect there is no:

Enter monitor0 service inner loop
new video service: monitor0
Create capturer
new encoder

The client therefore remains indefinitely at "Connected, waiting for image transmission".

Terminal still works while video is broken

While desktop video was still stuck, I opened the built-in RustDesk Terminal.

The same --server process successfully created the terminal service:

Creating new terminal service
Enter ts_... service inner loop
Creating new terminal 1
Opening PTY
Using shell: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Spawning shell process...
Terminal 1 opened successfully

This suggests that the RustDesk server process, authentication/connection path, and non-video services are still alive. The persistent failure appears specific to the video/monitor0 state.

Recovery experiment

During the stuck state, the process tree was approximately:

RustDesk.exe --service
└─ RustDesk.exe --server
   └─ RustDesk.exe --cm

Instead of restarting the Windows RustDesk service, I force-terminated only the existing:

RustDesk.exe --server

The --service process immediately spawned a new --server process.

Desktop access recovered immediately.

No Windows reboot and no restart of the parent RustDesk Windows service was required.

Therefore the persistent broken state appears to exist inside the --server process.

Full dump captured while stuck

Before terminating the stuck --server process, I captured a full-memory minidump of it using MiniDumpWriteDump.

The dump size is approximately:

796,838,895 bytes

The dump was taken while:

• desktop video was permanently stuck;
• reconnects were stuck at "Waiting for image";
• RustDesk Terminal still worked;
• the original --server process had not yet been restarted.

I would prefer not to upload the full dump publicly because it may contain session/configuration/clipboard data.

How to Reproduce

Unfortunately the failure is intermittent.

The general sequence is:

1. Use Windows -> Windows RustDesk remote desktop.
2. Hardware codec is enabled.
3. Intel QSV/MFX is selected for the controlled-side video encoder.
4. In my captured case RustDesk used:

DXGI capture
Intel MFX/QSV
H.265
hevc_qsv

5. Use the remote desktop normally for some time.
6. Eventually the image suddenly freezes.
7. FPS and bitrate fall to 0.
8. Disconnect and reconnect.
9. Connection succeeds but remains at "Connected, waiting for image transmission".
10. Built-in Terminal remains usable.
11. Kill only RustDesk.exe --server.
12. The RustDesk service respawns it and desktop video immediately works again.

I have experienced the same general failure on multiple Intel QSV machines.

Expected Behavior

A transient hardware encoder error should not permanently destroy the desktop video service.

Ideally RustDesk should recover from the failed hardware encoder automatically, for example by:

• recreating the hardware encoder;
• falling back to another codec/software encoder; or
• tearing down and recreating the monitor0 video service.

At minimum, disconnecting the failed session and reconnecting should create a fresh video service rather than leaving all future desktop sessions permanently stuck at "Waiting for image".

Operating system(s) on local (controlling) side and remote (controlled) side

Windows 10 Professional 19045.7663 -> Windows 10 Professional 19045.7663

RustDesk Version(s) on local (controlling) side and remote (controlled) side

1.4.9 -> 1.4.9

Screenshots

I forgot to take a screenshot.

Additional Context

Hardware

Affected controlled machines use Intel GPUs with Intel Quick Sync Video (QSV/MFX).

This has occurred on multiple QSV-equipped machines, not only one host.

The captured failure specifically used:

vendor: MFX
data_format: H265
encoder name: hevc_qsv

Additional Context

This appears closely related to #14662:

• desktop video works normally for some time;
• image suddenly freezes;
• reconnect remains permanently at "Waiting for image";
• restarting the controlled-side RustDesk process restores it.

The additional evidence here is that the failure was captured immediately after an Intel hevc_qsv encoder error, followed by an apparently incomplete monitor0 teardown/recreation.

There are also unrelated API/heartbeat/relay timeout messages in the server log, but those errors had already been occurring while desktop sessions were still able to start successfully. The failed reconnect itself also successfully establishes the RustDesk connection and starts non-video services, so network/relay failure does not appear to explain the persistent no-video state.

My current hypothesis is therefore:

QSV/FFmpeg runtime encoder failure
        ↓
video service stops making progress
        ↓
monitor0 is not fully cleaned up
        ↓
future desktop sessions cannot create a new monitor0
        ↓
permanent "Waiting for image"
        ↓
restart/respawn --server
        ↓
immediate recovery