ColorSpace override is not well defined

Video decoders can be given via configure a color space that is used to override the decoder detected color space, as defined in https://w3c.github.io/webcodecs/#output-videoframes.

That said:

The algorithm refers to [[active decoder config]] which is only assigned in constructor to null.
UA support is not very consistent when the provided VideoColorSpaceInit has undefined members.

Wrt point 2, locally testing on a VP9 stream, overriding the whole colorSpace to bt2020/pq/bt2020-ncl leads to consistent results in Chrome and Firefox.
Passing just the transfer with 'PQ' gives bt2020/pq/bt2020-ncl in Chrome but bt709/bt709/bt709 in Firefox.
According my reading of the spec, I would have expected null/pq/null.

I haven't checked funky color space combos, I wonder whether UAs may sanitise somehow these values to meaningful combos.
It would be good to improve the spec and give guidance if we cannot provide mandatory wording.

One possibility is to allow overriding just the fields that are defined in the configure config.
For instance, to support overriding only fullRange.

Wow, surprising. Thanks for catching! IIRC, [[active decoder config]] should be assigned after configure succeeds.
I'd guess Chrome is treating the config color space as undefined, but depending on what codec string you provided may pull the one there, and finally would look at the stream color space. In general any missing field makes the color space invalid in Chrome.

All in all, I agree it's underspecified.

@padenot

Some tests in web-platform-tests/wpt#60398.
It seems Chrome is indeed doing some kind of codec specific handling.

I could see that a website would only want to override one field to enable HDR or it might want to override all fields.
I assume that JS-provided color space should win over bit stream color space.

2. In general any missing field makes the color space invalid in Chrome.

Could you be more specific about Chrome's current algorithm?

@padenot, do you know what Firefox is doing in that area?

Sure, the order of operations seems to be:

Parse color space from codec string (some have defaults. e.g. vp9 default to 709): Ref
If config has any color space (even invalid), overwrite whatever we got from the codec string: Ref
For vp8, vp9 -- if bitstream has a colorspace, ignore it if config color space is fully valid: Ref

However, after some more sleuthing it seems step 3 breaks down in Chrome for AV1, H.264, and HEVC. For those codecs we will prefer the bitstream color space if it's valid. ISTR we tried to change this in the past for <video> and something exploded, but for WebCodecs we should change things such that we obey the config.

In step 2 we might want to throw an error when the config color space isn't fully specified, just to make it clear we won't use it, but I'm open to other ideas.

For 1, WebKit is not using the codec string so far.
For 2, WebKit updates the color space with whatever we have in the config, subfield per subfield (even if config color space is not fully defined).
For 3, webkit is overwriting what we get in the bitstream with the config (VP8/VP9/AV1 for now, H264/HEVC to follow).

In step 2 we might want to throw an error when the config color space isn't fully specified, just to make it clear we won't use it, but I'm open to other ideas.

WebKit is not doing that so far but can probably change if Firefox and Chrome already enforce this.
It is unclear to me from the spec whether encoders, when providing colorSpace need to provide fully specified color spaces.
https://w3c.github.io/webcodecs/#output-encodedvideochunks is not setting colorSpace at all, AIUI.

In step 2 we might want to throw an error when the config color space isn't fully specified

Is that web compatible? No UA is doing that AFAIUI.
From testing, it seems that all UAs are somehow reacting to partial color spaces, though not in a consistent way.
I uploaded a PR to cover this while we try to converge in that area.

To be clear I'm only suggesting we do that for a partially specified string, not one missing entirely. It would be a breaking change, but I suspect there aren't many cases out there.

If we get stricter there, it seems to beg the question whether VideoColorSpaceInit members should be all optional+nullable and null by default, at least as part of VideoDecoderConfig produced by encoders/consumed by decoders.
Maybe worth WG discussion time.

I think the spec used to be stricter, but there were some issues. See discussions on the PRs which relaxed this:

@youennf

I ran out of time to write this up as a PR today, but the consensus of the working group is that we should (in priority order):

If platform decoding results in a different color space, use the platform emitted color space. This is unavoidable on some platforms where hardware decoders may emit RGB, more esoteric formats during decoding, or may simply not support esoteric color space combinations.
Failing 1, if the config color space is fully specified, use it.
Failing 2, if the bitstream has fully specified color space, use it.
Failing 3, if the codec defines a default color space, use that color space. (E.g, VP9, AV1 define BT.709 as the default). Maybe we should choose a default for all codecs within the registry?

@youennf @padenot ^^^ to review

Thanks Dale for the sum up.

An alternative we might want to consider: expose whatever the config or bitstream signaled, partial or not, without trying to generate a fully-specified color space.
A few related notes below:

The case of RGB decoders makes sense to me.
The way you phrase it If platform decoding results in a different color space, use the platform emitted color space could allow any implementation to do whatever it wants.
Adding implementer guidelines like listing the RGB decoder exception is probably right.

AIUI, there is already a large support for partial color spaces: containers like webm, codecs like AV1/H264/H265, transport specific tools like webrtc color space header extension plus webcodecs webidl.
The web already supports creating and consuming partial color space video frames: https://jsfiddle.net/m8kx491q/.

We could simply state that config.colorSpace takes precedence over bitstream colorSpace but not check for fully specified at all.
This is the simplest approach, spec and implementation wise I think.
It also allows things like AV1 transcoding preservation of partial color space that would otherwise be impossible without tweaking the bitstream after transcoding.

The actual cost is that it leaves the handling of partial color space to downstream consumers, in particular renderers.
Rendering of partial color space video frames is an interesting question, which we need to answer anyway today whatever we decide for decoders.

As expressed during the video conf, the color space handling at encoder side is a bit light.
As per my reading of https://w3c.github.io/webcodecs/#output-encodedvideochunks, outputConfig.colorSpace is never set, which is probably a hole in the spec.

Similarly to the above, allowing partial color space might be the simple approach here.
Codecs whose bitstream can express partial color info like AV1 can preserve partial VideoFrame.colorSpace in-band. Other codecs like VP9 would populate outputConfig.colorSpace with the input frame's colorSpace whenever they cannot represent it in-band.

The case of RGB decoders makes sense to me. The way you phrase it If platform decoding results in a different color space, use the platform emitted color space could allow any implementation to do whatever it wants. Adding implementer guidelines like listing the RGB decoder exception is probably right.

I'm okay with tightening the language here, but I was trying to leave enough space for you to express the macOS/iOS CGColorSpace/PixelFormat limitations. Do you have a suggestion that leaves you with the necessary room?

AIUI, there is already a large support for partial color spaces: containers like webm, codecs like AV1/H264/H265, transport specific tools like webrtc color space header extension plus webcodecs webidl. The web already supports creating and consuming partial color space video frames: https://jsfiddle.net/m8kx491q/.

Yes, it's unfortunate that this ended up being the case. Ideally I'd have us reject such frames or fill in a default value (which is exactly what happens when rendering occurs).

We could simply state that config.colorSpace takes precedence over bitstream colorSpace but not check for fully specified at all. This is the simplest approach, spec and implementation wise I think. It also allows things like AV1 transcoding preservation of partial color space that would otherwise be impossible without tweaking the bitstream after transcoding.

The actual cost is that it leaves the handling of partial color space to downstream consumers, in particular renderers. Rendering of partial color space video frames is an interesting question, which we need to answer anyway today whatever we decide for decoders.

Rendering is why I'm against partial color spaces and where we've historically gotten the most complaints in <video>. In Chromium everything decoded is given a fully valid color space w/ BT.601 or BT.709 being the default when the color information is missing. I agree codifying what happens for frames created with partial color spaces (either directly or through decoding) is a good idea.

It's more work for Chromium to track a special WebCodecs partial color space separately from the frame's actual decoded color space, so I'd prefer we stamp the results of whatever algorithm we agree onto any decoded frames. I'm open to being convinced otherwise though.

I think it's less surprising for developers and less likely to result in interop issues if we codify an algorithm for filling in unspecified portions of the color space and reflect the result of that back on the VideoFrame.

As expressed during the video conf, the color space handling at encoder side is a bit light. As per my reading of https://w3c.github.io/webcodecs/#output-encodedvideochunks, outputConfig.colorSpace is never set, which is probably a hole in the spec.

Agreed, full-cycle-test.https.any.js verifies some properties of color spaces that we should codify.

Similarly to the above, allowing partial color space might be the simple approach here. Codecs whose bitstream can express partial color info like AV1 can preserve partial VideoFrame.colorSpace in-band. Other codecs like VP9 would populate outputConfig.colorSpace with the input frame's colorSpace whenever they cannot represent it in-band.

Even if we allow partial color spaces, we would need to choose precedence.

IIUC, your proposed steps would be:

If platform decoding results in a different color space, use the platform emitted color space (maybe only for RGB?)
Failing that, if the config specifies a color space, use it.
Failing that, if the bit stream specifies a color space, use it?
Failing all of the above, leave the color space unspecified.

Is that accurate? Do you have thoughts on what should happen during rendering then?

Is that accurate? Do you have thoughts on what should happen during rendering then?

That is mostly accurate, except for step 4 maybe.
Let's say we have an AV1 stream with color_description_present_flag set to true and all sub-fields are unspecified.
We would then expose a color space without any info in it, as per step 3.
Step 4 would kick in when color_description_present_flag is false.
It seems we cannot expose color_description_present_flag=false to the web without making VideoFrame.colorSpace nullable, which seems overkill.
Having a (per codec?) default makes sense to me there.

For rendering, hopefully we can just ignore partial color spaces.
I would go with the same rule, no matter whether frames come from JS or decoders.
A simple option is to follow pick color space and use sRGB for RGB and BT709 otherwise.
For the case where only fullRange is unspecified, using the rest of the color space and assuming studio range is a possibility.

Here is the complete comment with bt709 and rgb used for the matrix names in the Encoding Proposal:

According claude-generated https://jsfiddle.net/kp5hay47/, Chrome seems to be supporting partial color spaces for AV.1 encoding.
I can try to take a deeper look in that area if that helps the discussion.

This is an artifact of writing the color space one component at a time. The space is overall invalid, but frames can be created manually with invalid spaces. This does point in a direction where we might find agreement though:

VideoFrameBufferInit Proposal

No changes: partial spaces will continue to be accepted, but have consequences during encoding or rendering.

Decoding Proposal

If platform decoding transforms the color space (e.g., hardware decode to RGB or range/matrix conversion), use the platform-emitted color space.
Otherwise, between the config color space and the bitstream color space, prefer the most specified color space (the one with the most non-null fields, breaking ties in favor of the config color space):
- If the selected color space is fully specified, use it.
- If the selected color space is not fully specified (partial or all null):
  - If platform decoding requires a fully specified color space, use codec-specific defaults, or REC709 (bt709/bt709/bt709/limited) if no codec-specific default is defined.
  - Otherwise, use the selected color space as-is for the emitted frame.

Encoding Proposal

Encoders may modify the color space matrix and range as needed to meet codec requirements. If the matrix or range are missing (or no color space is provided), limited-range bt709 will be assumed for YUV formats and full-range rgb for RGB formats.
The color space emitted within EncodedVideoChunkMetadata should be set to the color space reported by the encoder if the encoder specifies one (ignoring unspecified values returned by platform encoders). Otherwise, it should be the last known color space before the frame is handed off to the encoder.

Rendering Proposal

If a color space is missing or only partially specified, use REC709 (bt709/bt709/bt709/limited) for YUV formats and sRGB (bt709/iec61966-2-1/rgb/full) for RGB formats.

I appreciate being a leaf in the wind given the precedent for partial color spaces. Here's a brain dump of all my knowledge in this space to draft your own proposals from:

Codecs

H264, HEVC, and AV1 have expressive metadata in the bitstream for color information, with explicit values for unspecified. The bitstreams also have flags for whether color information is present at all.
VP9 has limited color support in the bitstream; exposing just a range (always present) and a single unified color space field (fully unknown or fully specified).
VP8 is technically always BT.601, but has 1-bit that ISTR being used as a range flag.

Decoding

H264, HEVC, AV1 decoders may inherit unspecified color space fields from the config or the bitstream.
Hardware decoders or even post-software-decode-prep work done for <video> efficiency may change color spaces entirely. In practice I've only seen matrices and range change while primaries and transfer remain the same.
In Chromium today, hardware decoders will always burn in a fully specified color space -- assuming BT.709 if config/bitstream are not fully specified. We used to guess, but now no guessing is done since we've had bugs filed for both sides of a possible guess.
Hardware resources may require fully specified color spaces.

VideoFrame construction

Frames may be constructed directly or through CanvasImageSources with unspecified color space fields. Maybe only <video> sources can possibly end up with unspecified fields though?

Encoding

The encoder (or UA pre-encode pipeline) may change the color space to satisfy codec requirements (e.g., RGB -> YUV conversion). Ideally primaries, transfer, and any HDR metadata are preserved though -- this was not the case until recently in Chromium. This means the post-encode emitted color space may be different than an input frame's.
For H264, HEVC, and AV1, if an encoder can accept the frame directly, the parts of a frame's color space that are specified will be passed in. Generally the primaries/transfer/matrix/range can be specified individually, but this is not always the case for OS encoders.
Even when color information is given to the platform, some platform encoders (e.g., MediaFoundation) may still fill in explicit unspecified values within the bitstream for the color space information. (Contrary to your point in ColorSpace override is not well defined #940 (comment))
Even if an OS encoder writes unspecified into the bitstream we probably don't want to pick that up. We should propagate the known color space at the time of hand-off to the OS encoder (or if the OS encoder explicitly returns a different specified color space, use that).

Rendering

A color space is required for rendering.
In Chromium when one is missing, we'll assume BT.709 or sRGB depending on the matrix.

And of course after I posted that I realized we have one other wrinkle: The WebCodecs spec only specifies the most popular color spaces values. It is not fully expressive. We'll either need to make it more expressive (hard to be future proof) or use the same trick we did with format.

Gemini suggests this additional text:

Internal vs. WebIDL Color Spaces

Like VideoFrame.format (which returns null in JS when the underlying pixel format isn't in the WebIDL enum), a VideoFrame has an internal color space on its underlying media resource that may be more expressive than the WebIDL enums (e.g., full H.273 code points).
Any internal color space field that does not map to a WebCodecs WebIDL enum is exposed as null on VideoFrame.colorSpace, but decoding, rendering, and encoding decisions operate on the internal color space. "Unspecified" and "partially specified" below refer to the internal color space (e.g., H.273 unspecified or omitted in VideoFrameBufferInit), not unmapped WebIDL enums.

Clarify color space handling across decoding, encoding, and rendering

I went ahead and wrote a PR for the above which we can discuss more specifically. Thanks in advance!

Thanks for moving forward and sorry for the delay here.
I did some improvement to WebKit support and experimented a bit as well, see web-platform-tests/wpt#63157 for partial color space encode/decode tests.

Overall, I think the proposal is going in the right direction, some additional comments:

Decoding Proposal

If platform decoding transforms the color space (e.g., hardware decode to RGB or range/matrix conversion), use the platform-emitted color space.

OK

Otherwise, between the config color space and the bitstream color space, prefer the most specified color space (the one with the most non-null fields, breaking ties in favor of the config color space):

To keep things simple and unsurprising, I would always pick the color space provided by the web page, if one is provided.
The case I am not totally sure is colorSpace: { } vs. colorSpace: undefined.

If no colorSpace config is provided and the bit stream has some unspecified values, I am unsure what we should mandate.

 * If platform decoding requires a fully specified color space, use codec-specific defaults, or REC709 (`bt709`/`bt709`/`bt709`/`limited`) if no codec-specific default is defined.

Do you know which platforms mandate a fully specified color space?

Encoding Proposal

Encoders may modify the color space matrix and range as needed to meet codec requirements. If the matrix or range are missing (or no color space is provided), limited-range bt709 will be assumed for YUV formats and full-range rgb for RGB formats.

The color space emitted within EncodedVideoChunkMetadata should be set to the color space reported by the encoder if the encoder specifies one (ignoring unspecified values returned by platform encoders). Otherwise, it should be the last known color space before the frame is handed off to the encoder.

I would tend to design it as:

Encoders MAY convert a VideoFrame into a new VideoFrame just before the encoding step (hopefully this is the exception case, just like for decoding).
The colorSpace of the VideoFrame fed to the encoding step is the one exposed with EncodedVideoChunkMetadata, irrespective of the color space actually encoded in the bitstream.
For codecs that support encoding of individual color space fields, the encoding step SHOULD encode each color space field faithfully in the bitstream.

I wonder whether the spec should provide guidelines on how to handle color space changes mid-stream.
WebKit generates a new key frame/new config in that case.

Any internal color space field that does not map to a WebCodecs WebIDL enum is exposed as null on VideoFrame.colorSpace

null already means unspecified color space. The UA has to pick bt709 as per the rendering rule we agree on.
For internal color space fields that do not map, the UA knows how to render and will not pick bt709.
I would tend to expose these values as other.

Frames may be constructed directly or through CanvasImageSources with unspecified color space fields. Maybe only <video> sources can possibly end up with unspecified fields though?

For frames created from a regular video element (MSE, HLS...), I would be ok mandating fully valid color spaces, typically the one used by the UA to render the content.
I am less sure about the VideoTrackGenerator-fed-video case.

see web-platform-tests/wpt#63157 for partial color space encode/decode tests.

Seems fine to me % vp8, vp9 (which can't represent partial color spaces in the bitstream). Chrome doesn't run web tests with hardware encoding, and our software encoders for H.264/AV1 should be writing the bitstream values.

To keep things simple and unsurprising, I would always pick the color space provided by the web page, if one is provided. The case I am not totally sure is colorSpace: { } vs. colorSpace: undefined.

How do you resolve #940 (comment) then; I.e., where the bitstream may be more accurately specified than the config (for example, when a config generated by VideoEncoder or a demuxer has null for an unmapped H.273 value that is present in the bitstream)? Are you proposing we allow other as an input value? Applications would have to modify the config based on whether the color space can be fully specified in WebCodecs IDL values in order to get accurate rendering.

In WebIDL, all members of VideoColorSpaceInit default to = null, so colorSpace: {} is indistinguishable from {primaries: null, transfer: null, matrix: null, fullRange: null}. (As you found in web-platform-tests/wpt#63157, existing tests like full-cycle-test.https.any.js actually pass config.colorSpace = {} expecting the bitstream color space to be used.)

Preferring the most specified color space (breaking ties in favor of config.colorSpace) solves this naturally without any special case for {}:

colorSpace: {} (0 specified fields) loses to any bitstream with specified fields, matching colorSpace: undefined.
A fully specified config.colorSpace (4 fields) always wins over the bitstream.
If we aren't merging field-by-field, letting a 1-field config.colorSpace (e.g. {fullRange: true}) wipe out a 4-field HDR bitstream (bt2020/pq/bt2020-ncl) would cause the frame to fall back to BT.709 at render time.
With other we'd be back to trying to merge the two color spaces if the bitstream has some unspecified fields as well.

If no colorSpace config is provided and the bit stream has some unspecified values, I am unsure what we should mandate.

Unless the decoder requires a fully specified color space, we can expose the partial bitstream color space as-is and only mandate a fallback when rendering. I think sRGB and BT.709 are sensible in both cases. Trying to piece together a guessed color space is another option, but seems surprising per our previous discussions.

Do you know which platforms mandate a fully specified color space?

Android uses a bundled color space similar to VP8/VP9 (KEY_COLOR_STANDARD bundles primaries and matrix), Windows has expressiveness issues similar to our WebIDL problems, and even macOS/iOS requires that the range be specified due to the kCVPixelFormatType limitation. Chromium today burns in a fully specified color space for all hardware decoders since it's required by our GPU texture/data abstractions (SharedImage)

Encoders MAY convert a VideoFrame into a new VideoFrame just before the encoding step (hopefully this is the exception case, just like for decoding).

It's actually a very common one, e.g., how do you encode RGB frames from canvas? (Note also that some hardware encoders accept RGB surfaces and convert to YUV internally during encoding, so EncodedVideoChunkMetadata needs to reflect the post-encode color space if the encoder changed it.)

For codecs that support encoding of individual color space fields, the encoding step SHOULD encode each color space field faithfully in the bitstream.

Hardware encoders seem to almost never do this unfortunately. SHOULD allows wiggle room though, so seems okay.

I wonder whether the spec should provide guidelines on how to handle color space changes mid-stream. WebKit generates a new key frame/new config in that case.

Yes, Chrome does as well, but hardware encoders are a different story. Some versions of Android in particular treat key frame generation as a request rather than a requirement.

null already means unspecified color space. The UA has to pick bt709 as per the rendering rule we agree on. For internal color space fields that do not map, the UA knows how to render and will not pick bt709. I would tend to expose these values as other.

Per the above, I think introducing other would just compound our problems instead of solving any. I.e., how does other work as input? When decoding, what good is other to the user?

Note that if the rendering rule is defined against the media resource's internal color space (just like VideoFrame.format === null still renders using the internal pixel format), an unmapped H.273 value is still fully specified internally and will not fall back to bt709 even though the JS-exposed VideoFrame.colorSpace field is null.

For frames created from a regular video element (MSE, HLS...), I would be ok mandating fully valid color spaces, typically the one used by the UA to render the content. I am less sure about the VideoTrackGenerator-fed-video case.

I'd like to try and avoid special casing <video> versus WebCodecs if we can; it's not impossible, but will result in a parallel metadata system just for WebCodecs in the Chromium code base.

How do you resolve #940 (comment) then; I.e., where the bitstream may be more accurately specified than the config

If the web page is not sure of the demux color space, it can check what the bitstream color space is (by decoding with config.colorSpace: undefined). Then it could either continue with what the bitstream says.
Or it can generate its own colorSpace and reconfigure its decoder.
I would expect this case to be rare enough we do not need to optimise it.

colorSpace: {} is indistinguishable from {primaries: null, transfer: null, matrix: null, fullRange: null}

Right, but this is different from colorSpace: undefined, which tells the UA to use the bitstream color space.
I am ok treating colorSpace: {} equivalent to undefined.

Unless the decoder requires a fully specified color space, we can expose the partial bitstream color space as-is

Right, that is also what I think, it aligns well with WebCodecs being low level. But it has a cost.
That for instance requires UA to parse bitstream to get the exact bitstream color space.
macOS HW decoders will guess a full color space if the bitstream provides a partial one.
This guessed color space is probably what video elements will actually use for rendering HLS content.
I guess we can formalise it as HLS video elements do their decoding with a custom config.colorSpace override when bitstream color space is partial.

I think introducing other would just compound our problems instead of solving any. I.e., how does other work as input? When decoding, what good is other to the user?

A VideoFrame constructor should throw if a color space entry is set to other, only UA created VideoFrames could have that value.

other is useful to the web page as it tells that rendering/conversion will be done properly by the UA.
Web page will not assume that BT709 is what will be used.
Web page does not need to try providing a config.colorSpace to try fixing a partial color space.

Clarify color space handling across decoding, encoding, and rendering

I've updated the proposal to adopt your earlier suggestion for fullRange (preserving primaries, transfer, and matrix when only fullRange is unspecified, and defaulting fullRange to limited/false for YUV and full/true for RGB). I think this is reasonable and solves the most common foot-guns with the most-specified approach.

Decoding:
- If the decoder transforms the color space during decode, use the transformed space.
- Otherwise, select the most specified color space between config.colorSpace and the bitstream based on primaries, transfer, and matrix (breaking ties in favor of config.colorSpace). This naturally treats colorSpace: {} as equivalent to undefined, and ensures a 3-field config.colorSpace (where fullRange was omitted) still overrides a 4-field bitstream.
- If primaries, transfer, or matrix is unspecified and the underlying decoder requires (or fills in, like macOS/Chromium HW decoders) a complete color space, fall back to the codec default or sRGB (RGB) / BT.709 (other formats). Similarly, if the decoder requires range to be specified (e.g., kCVPixelFormatType on macOS), default an unspecified fullRange to true for RGB and false otherwise.
Encoding:
- Encoders may modify matrix and fullRange to satisfy codec requirements (assuming full-range rgb for RGB formats and limited-range bt709 otherwise when unspecified).
- outputConfig.colorSpace is populated from the encoder-emitted color space if specified, or else the frame's internal color space after any modifications above.
Rendering:
- If the frame's internal primaries, transfer, and matrix are specified, render using that color space (treating an unspecified fullRange as true for RGB and false otherwise). Only fall back to sRGB (for RGB) or BT.709 (otherwise) when primaries, transfer, or matrix is unspecified.

Can you help me understand your remaining concerns with the "most-specified" approach? Are you trying to avoid having the UA parse the bitstream? AFAICT the only way to avoid that is to either (a) allow OS decoders to do whatever they want or (b) add all H.273 code points to the spec. The following would remain unchanged either way:

We'd still need to let OS decoders carve out an exception for transforms.
We'd still need the default behavior for range, rendering, and encoding transforms to be specified.

Since (a) will always be true and outside of our control for high-level decoders like VideoToolbox and MediaCodec, I took a stab at what embracing (a) would look like here: #957 -- I still think there's value in specifying the ideal bitstream level behavior for low level decoders (D3D11, VAAPI) though, so I kept some of the bitstream parsing, but left it up to the [[codec implementation]]

Can you help me understand your remaining concerns with the "most-specified" approach?

The main concern is predictability. With "most-specified", the same config can mean "override" for one stream and "ignored" for another, depending on what the bitstream contains. The page can't tell which without parsing the bitstream itself.

As an example, a web application could feed BT601/BT709/null to a VP9 encoder and end up with BT601/BT601/BT601 decoded frames even though the web page asked for BT601/BT709/null in the config.
This can get even more complex for the web page to reason about if we go with null meaning either unspecified or not-representable-by-webcodecs.
Again, the only way for the web page to know what is going on would be to parse the bitstream.

With "config always wins", we would have something like:

If the page provides a non-empty colorSpace, frames use it, with your fullRange defaulting for unspecified fields.
If the page provides none (undefined or {}), the decoder uses the bitstream color space or the codec default. That also covers the web-compat case.
If a specified field on the frame differs from the config, the decoder transformed it, and the page can rely on that.

This is predictable and also aligns with WebCodecs trying to be a thin layer on top of codec implementations.

Are you trying to avoid having the UA parse the bitstream?

It would be nice but it is probably a lost cause.
It would be nice if web pages would not have to parse bitstreams themselves.

We'd still need to let OS decoders carve out an exception for transforms.

We'd still need the default behavior for range, rendering, and encoding transforms to be specified.

Agreed on both, whatever we choose. "most-specified" is also shimable from "config always wins".

@Vanilagy

Okay, I can see the case you're arguing more clearly now.

Agreed on both, whatever we choose. "most-specified" is also shimable from "config always wins".

To clarify, you mean that the OS transform exception also carves out an exception for when [[codec implementation]] chooses more specified values from the bitstream? If so, isn't that just what the current text says?

@Vanilagy given you maintain one of the largest WebCodecs' projects, do you have opinions on this issue?

This is a lot to read so I only skimmed over it. Haven't had too many issues with this, although I also haven't thought about which color space should win if both the decoder metadata and the bitstream specify a color space. My gut tells me the bitstream should be king, and color space metadata provided to the decoder is simply supplementary if color space information is not contained (or partial) in the bitstream.

I recently had to smooth over this oddity in Chrome:

This was on a file that only defined the matrix and fullRange fields.

@padenot

Thanks! That's https://crbug.com/532860177 and repros in Safari too. It's due to VideoToolbox emitting a fully specified color space on the decoded CVPixelBuffer that the UA assumes has precedent over the config. It's a relevant case to discuss here since it's an OS decoder indicating a color space where the UA doesn't know if the OS just tagged its own guess or has done color transformation.

Without internal knowledge, none of the solutions discussed here would fix this though. We need to special case that VideoToolbox won't color convert (only range-convert IIUC) and then prefer the config color space. The UA also needs to ensure they select the pixel format which matches range on macOS/iOS to avoid further post-decode config munging.

I think we're all in agreement we should prefer the config color space where possible. The remaining disagreement is whether or not we only allow VideoDecoderConfig.colorSpace values of null and {} to tell the UA to use the bitstream color space. The alternative being that the bitstream color space can also be used if it's more specified than the config value.

@padenot any thoughts to help break one way or the other?

Navigation Menu

Repository navigation

ColorSpace override is not well defined #940

2 remaining items

VideoFrameBufferInit Proposal

Decoding Proposal

Encoding Proposal

Rendering Proposal

Codecs

Decoding

VideoFrame construction

Encoding

Rendering

Internal vs. WebIDL Color Spaces

Decoding Proposal

Encoding Proposal

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ColorSpace override is not well defined #940

Description

Activity

dalecurtis commented on Jun 4, 2026

youennf commented on Jun 16, 2026

dalecurtis commented on Jun 16, 2026

youennf commented on Jun 22, 2026

youennf commented on Jul 7, 2026

dalecurtis commented on Jul 7, 2026

youennf commented on Jul 8, 2026

dalecurtis commented on Jul 8, 2026

dalecurtis commented on Sep 3, 2026

youennf commented on Sep 7, 2026

dalecurtis commented on Sep 9, 2026

youennf commented on Sep 10, 2026

2 remaining items

dalecurtis commented on Sep 15, 2026

VideoFrameBufferInit Proposal

Decoding Proposal

Encoding Proposal

Rendering Proposal

Codecs

Decoding

VideoFrame construction

Encoding

Rendering

dalecurtis commented on Sep 15, 2026

Internal vs. WebIDL Color Spaces

dalecurtis commented on Sep 30, 2026

youennf commented on Oct 1, 2026

Decoding Proposal

Encoding Proposal

dalecurtis commented on Oct 1, 2026

youennf commented on Oct 2, 2026

dalecurtis commented on Oct 6, 2026

youennf commented on Oct 6, 2026

dalecurtis commented on Oct 6, 2026

Vanilagy commented on Oct 7, 2026

dalecurtis commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions