fftools/ffmpeg_mux: fix stream-copied audio dropped with -ss and -output_ts_offset by LTe · Pull Request #685 · jellyfin/jellyfin-ffmpeg

LTe · 2026-04-10T10:16:31Z

Changes

When using -copyts with output -ss, timestamps are zeroed to ~0 instead of
being preserved at the seek position.

Without -copyts, output -ss 120 correctly produces output starting at 0.
With -copyts but without output -ss, timestamps are correctly preserved at ~120.
However, -copyts combined with output -ss 120 incorrectly zeros timestamps to ~0.

Root cause

Three places unconditionally subtract start_time from timestamps regardless
of -copyts:

of_streamcopy() subtracts start_time from stream-copied packet timestamps
The encoder filter chain uses ts_offset = start_time to shift encoded output
back to zero
check_recording_time() in the encoder compares frame PTS against
recording_time without accounting for the preserved timestamps

With -copyts, the demuxer preserves original timestamps (~120s). Subtracting
start_time zeros them, which is correct without -copyts (where the demuxer
already reset to ~0) but wrong with -copyts.

Fix

Three changes, all gated on copy_ts:

fftools/ffmpeg_mux.c: Skip start_time subtraction in of_streamcopy()
when copy_ts is set. Stream-copied packets keep their original PTS.
fftools/ffmpeg_mux_init.c: Set filter ts_offset = 0 when copy_ts is
set. With -copyts, input frames already carry the real PTS, so the trim
filter sees them directly without a round-trip shift.
fftools/ffmpeg_enc.c: In check_recording_time(), compare against
recording_time + start_time when copy_ts is set. This matches the existing
stream-copy check at ffmpeg_mux.c:469 and prevents -t from stopping
encoding immediately when frame PTS is at the seek position.

Both encoded and stream-copied packets stay at their original PTS throughout the
pipeline, so the interleaver sees no DTS gap between streams.

Tests

8 new FATE tests covering all flag combinations:

HLS tests (hlsenc.mak, framecrc):

fate-streamcopy-extaudio-nocopyts: -output_ts_offset without -copyts
fate-streamcopy-extaudio-plain: output -ss only
fate-streamcopy-extaudio-partial-offset: partial -output_ts_offset
fate-streamcopy-extaudio-copyts: -copyts with output -ss
fate-streamcopy-extaudio-copyts-interleave: 30s seek with -copyts

Encoder tests (enc-copyts.mak, transcode md5+framecrc):

fate-enc-copyts-mpeg4-ss: input -ss only
fate-enc-copyts-mpeg4-ss-copyts: input -ss + -copyts
fate-enc-copyts-mpeg4-ss-outss-copyts-offset: explicit -output_ts_offset

All tests use built-in encoders only (mpeg4, mp2fixed). Full FATE suite
passes with --assert-level=2.

Issues

https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/22765
jellyfin/jellyfin#16580

gnattu · 2026-04-10T10:27:46Z

First of all, please use quilt and submit as a patch, as this is how we adding changes to ffmpeg.
Secondly, this changes a fundamental behavior of ffmpeg muxer which, IMO, should have more concrete testing to be merged.

nyanmisaka · 2026-04-10T10:30:16Z

@LTe I just saw your PR on the ffmpeg repo and ran the fate tests on version 8.1, which is causing some failures. Is this expected?

Another question is, how do you plan to probe ffmpeg to see if this fix is included? We do not force users to use jellyfin-ffmpeg in their custom builds, or users may be using an older version of jellyfin-ffmpeg. Therefore, we expect the ffmpeg command-line to be backward compatible.

--- ./tests/ref/fate/matroska-avoid-negative-ts 2026-03-17 02:13:05.000000000 +0800
+++ tests/data/fate/matroska-avoid-negative-ts  2026-04-10 18:17:42.192301786 +0800
@@ -1,5 +1,5 @@
-fb4e7a969ef65f61c4c45d5976188aa2 *tests/data/fate/matroska-avoid-negative-ts.matroska
-973080 tests/data/fate/matroska-avoid-negative-ts.matroska
+3718a993550368868eb5d8eb1ca9bb19 *tests/data/fate/matroska-avoid-negative-ts.matroska
+902848 tests/data/fate/matroska-avoid-negative-ts.matroska
 #extradata 0:       22, 0x2885037c
 #tb 0: 1/1000
 #media_type 0: video
@@ -11,31 +11,20 @@
 #codec_id 1: mp3
 #sample_rate 1: 44100
 #channel_layout_name 1: mono
-0,        -37,         43,       40,     9156, 0xe5bd034a
-1,          0,          0,       26,      417, 0x7198c15e
-0,          3,          3,       40,     1740, 0x29ac4480, F=0x0
-1,         26,         26,       26,      417, 0x3c67c32d
-0,         43,        123,       40,     3672, 0x98652013, F=0x0
-1,         52,         52,       26,      417, 0x8c24b1ca
-1,         78,         78,       26,      417, 0x6ee576b7
-0,         83,         83,       40,     2532, 0xa2c42769, F=0x0
-1,        104,        104,       26,      417, 0x407603db
-0,        123,        203,       40,     1728, 0xae823d3b, F=0x0
+1,          0,          0,       26,      417, 0x3c67c32d
+1,         26,         26,       26,      417, 0x8c24b1ca
+1,         52,         52,       26,      417, 0x6ee576b7
+1,         78,         78,       26,      417, 0x407603db
+1,        104,        104,       26,      417, 0xcf2804d2
 1,        130,        130,       26,      417, 0xcf2804d2
 1,        156,        156,       26,      417, 0xcf2804d2
-0,        163,        163,       40,     1028, 0x286ac52a, F=0x0
 1,        182,        182,       26,      417, 0xcf2804d2
-0,        203,        283,       40,     1916, 0xd378899e, F=0x0
-1,        208,        208,       26,      417, 0xcf2804d2
+1,        209,        209,       26,      417, 0xcf2804d2
 1,        235,        235,       26,      417, 0xcf2804d2
-0,        243,        243,       40,     1168, 0x424e12cf, F=0x0
 1,        261,        261,       26,      417, 0xcf2804d2
-0,        283,        363,       40,     1660, 0x5cec156c, F=0x0
-1,        287,        287,       26,      417, 0xcf2804d2
-1,        313,        313,       26,      417, 0xef163d04
-0,        323,        323,       40,     1004, 0xac0dce29, F=0x0
-1,        339,        339,       26,      417, 0x2a009b3a
-0,        363,        443,       40,     3008, 0x0fc798bf, F=0x0
-1,        365,        365,       26,      417, 0xbedccb9d
-1,        365,        365,       26,      417, 0x2214be3f
-1,        391,        391,       26,      417, 0x8953b878
+1,        287,        287,       26,      417, 0xef163d04
+1,        313,        313,       26,      417, 0x2a009b3a
+1,        339,        339,       26,      417, 0xbedccb9d
+1,        339,        339,       26,      417, 0x2214be3f
+1,        365,        365,       26,      417, 0x8953b878
+1,        391,        391,       26,      417, 0x9fcdb36e
Test matroska-avoid-negative-ts failed. Look at tests/data/fate/matroska-avoid-negative-ts.err for details.
make: *** [tests/Makefile:323: fate-matroska-avoid-negative-ts] Error 1
make: *** Waiting for unfinished jobs....

LTe · 2026-04-10T11:28:23Z

First of all, please use quilt and submit as a patch, as this is how we adding changes to ffmpeg.

Secondly, this changes a fundamental behavior of ffmpeg muxer which, IMO, should have more concrete testing to be merged.

Noted, I will prepare patch with quilt

LTe · 2026-04-10T11:30:48Z

@LTe I just saw your PR on the ffmpeg repo and ran the fate tests on #664, which is causing some failures. Is this expected?

No this is not expected. I was testing using local generated files. I will try to run full suite with external assets.

Another question is, how do you plan to probe ffmpeg to see if this fix is included? We do not force users to use jellyfin-ffmpeg in their custom builds, or users may be using an older version of jellyfin-ffmpeg. Therefore, we expect the ffmpeg command-line to be backward compatible.

In case it will be merged to upstream we can just check version.
In case when it will stays in jellyfin-ffmpeg forever we can do something like ffmpeg -streamcopy_ts_fix 1 -f lavfi -i nullsrc=s=1x1:d=1 -f null and it should fail in upstream but we will handle that here adding no-op flag.

nyanmisaka · 2026-04-10T11:35:57Z

@LTe I just saw your PR on the ffmpeg repo and ran the fate tests on #664, which is causing some failures. Is this expected?

No this is not expected. I was testing using local generated files. I will try to run full suite with external assets.

Please run the fate tests as described in the link: https://ffmpeg.org/fate.html

In addition, I noticed that the test items here are different from those added in the ffmpeg repo. Is it because they haven't been synchronized yet?

LTe · 2026-04-10T11:56:22Z

@nyanmisaka The matroska-avoid-negative-ts failure is fixed. The issue was that the threshold adjustment triggered for negative output_ts_offset values too. With output_ts_offset = -60ms, subtracting a negative value raised the discard threshold instead of lowering it, causing valid video packets to be dropped.

In addition, I noticed that the test items here are different from those added in the ffmpeg repo. Is it because they haven't been synchronized yet?

The test variable wiring differs because jellyfin-ffmpeg 7.1.3's hlsenc.mak doesn't have the FATE_HLSENC_LAVFI section that upstream 8.x has. The actual test commands, source generation, and reference files are identical. We can add the FATE_HLSENC_LAVFI infrastructure to the quilt patch if you prefer exact parity, but it would mean backporting unrelated upstream test changes.

I can also prepare pull request against version 8.0, but then it will be to your repository I think. At least until #664 will be merged.

nyanmisaka · 2026-04-10T12:37:10Z

@nyanmisaka The matroska-avoid-negative-ts failure is fixed. The issue was that the threshold adjustment triggered for negative output_ts_offset values too. With output_ts_offset = -60ms, subtracting a negative value raised the discard threshold instead of lowering it, causing valid video packets to be dropped.

In addition, I noticed that the test items here are different from those added in the ffmpeg repo. Is it because they haven't been synchronized yet?

The test variable wiring differs because jellyfin-ffmpeg 7.1.3's hlsenc.mak doesn't have the FATE_HLSENC_LAVFI section that upstream 8.x has. The actual test commands, source generation, and reference files are identical. We can add the FATE_HLSENC_LAVFI infrastructure to the quilt patch if you prefer exact parity, but it would mean backporting unrelated upstream test changes.

I can also prepare pull request against version 8.0, but then it will be to your repository I think. At least until #664 will be merged.

Let's leave it as it is. I want to keep this open for a while, so we can wait for upstream ffmpeg devs to leave some feedback.

Regarding 8.1, you only need to update your PR in the ffmpeg repo, and I will update my branch accordingly once this PR is merged.

LTe · 2026-04-11T12:04:10Z

Your recent changes to the PR in ffmpeg repo caused fate tests to fail. The MPEG4 encoder seems to output different results on different machines.

Damn, I guess generating content in FATE specs is not really good idea. I guess I will try use ffmpeg samples — this should be consistent.

Add patch that sets filter ts_offset to 0 and skips start_time subtraction in of_streamcopy() when copy_ts is set. Also fixes check_recording_time() to account for start_time with -copyts. This avoids needing -output_ts_offset or -max_interleave_delta 0, fixing HLS segments missing audio on seeks beyond 10s.

gnattu · 2026-04-12T15:38:56Z

+    int64_t end_time = of->recording_time;
+
+    if (copy_ts && of->start_time != AV_NOPTS_VALUE)
+        end_time += of->start_time;


One thing I'm not very sure if it is correct: do_subtitle_out() does pts -= of->start_time; before calling this helper if the start_time is non zero. Which means the ts being passed in is always 0 based and the end time now will be far beyond the original subtitle end time, if I understand it correctly. Although Jellyfin current does not use subtitle path for this, but current behavior will very likely break that use case and we have a lot of users use jellyfin ffmpeg for general use cases.

Another problem here is that this addition is not bound checked so we do have potential of signed overflow (although rare). The original logic does the bound check for of->recording_time before av_compare_ts but I think we added another potential of overflow here, because end_time is not only inheriting the recording_time value, it would also add another value on top of it.

Maybe not that rare, I just realized that without -t, recording_time is intentionally initialized to INT64_MAX, which does cover a huge use case.

LTe · 2026-04-14T12:06:49Z

There is a workaround in user space with drop filter. Therefore custom patch is not needed. There is still a bug but I think it should be fixed in upstream.

https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/22765#issuecomment-36410

LTe mentioned this pull request Apr 10, 2026

Fix A/V desync when resuming HLS with video transcode + audio copy jellyfin/jellyfin#16580

Merged

LTe force-pushed the fix/streamcopy-audio-seek branch from 00a6477 to 6a1d691 Compare April 10, 2026 12:07

This comment was marked as outdated.

Sign in to view

nyanmisaka reviewed Apr 10, 2026

View reviewed changes

Comment thread debian/patches/0096-add-fate-tests-for-streamcopy-extaudio-seek.patch Outdated

LTe force-pushed the fix/streamcopy-audio-seek branch 3 times, most recently from 69ee8b3 to 7e15e3d Compare April 11, 2026 10:54

nyanmisaka reviewed Apr 11, 2026

View reviewed changes

Comment thread debian/patches/0095-preserve-timestamps-with-copyts-and-output-ss.patch Outdated

This comment was marked as outdated.

Sign in to view

LTe force-pushed the fix/streamcopy-audio-seek branch from 7e15e3d to ca843e4 Compare April 11, 2026 19:17

gnattu reviewed Apr 12, 2026

View reviewed changes

LTe closed this Apr 14, 2026

Uh oh!

Conversation

LTe commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Tests

Uh oh!

gnattu commented Apr 10, 2026

Uh oh!

nyanmisaka commented Apr 10, 2026

Uh oh!

LTe commented Apr 10, 2026

Uh oh!

LTe commented Apr 10, 2026

Uh oh!

nyanmisaka commented Apr 10, 2026

Uh oh!

LTe commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nyanmisaka commented Apr 10, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as outdated.

LTe commented Apr 11, 2026

Uh oh!

gnattu Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gnattu Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

gnattu Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

LTe commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LTe commented Apr 10, 2026 •

edited

Loading

LTe commented Apr 10, 2026 •

edited

Loading

gnattu Apr 12, 2026 •

edited

Loading