Repository navigation
[meta] PROM-35 Remote Write (PRW2.0) Stability #16944
Description
Context
This is an umbrella issue collecting things we plan to implement for the stable Prometheus Remote Write 2.0 (PRW2.0) protocol and Prometheus implementation; also described by the PROM-35 proposal. Superseeds #1310 and #14360
Anyone is welcome to comment or DM me, @cstyan or @saswatamcode to add/update/remove things from this list!
Acceptance Criteria
This is issue will be closed when we will be able to mark PRW2.0 as stable and when we can start the official process of planned deprecation and switching the default PRW method to V2 instead of V1. That "stability", depends in general, on:
a) The acceptable state (functionality and performance) of the Prometheus receive and export (write) Remote Write 2 implementation.
b) Some adoption for (at least) receiving in the ecosystem, by systems like Cortex, Thanos, Mimir and/or vendors.
Tasks
List of tasks towards stability.
Protocol
- Decide what to do with "Timestamp on exemplar is MUST" @cstyan
- Decide on delta type improvement (specifically CT/ST per sample addition) @fionaliao @carrieedwards
- NewRelic dev asked in the past what to do with the WriteResponse when the parsing is async. Extend the protocol to cover this case (e.g. "Written" header with "N/A" value or so) (breaking change)
- Consider the best way to allow exemplar per sample and per series handling
Prometheus Implementation
- Ensure type-and-unit feature works with RW ([meta] PROM-39 type-and-unit-labels stability #16610).
- Document behaviour (remote write 2.0 - decide whether addition of metadata should count towards max samples in write request #14407)
- Prometheus uses the official RW client on receive (thanks to @pipiland2612!)
- Ensure RW new features works on agent mode (remote write 2.0 - test for agent mode with metadata-wal-records and type-and-unit features #13483).
- STs are working with RW sending (ST: Prometheus propagates ST in Remote-Write 2.0 #14220)
- PRW1 metadata parity ([RW2] metadata.send feature parity with v1 #17191) (in progress [RW2.0] Enable fast metadata look up for Remote Write 2.0 #17436)
- Decide on PRW2 metadata instrumentation semantics (storage/remote: ensure metadata instrumentation make sense for PRW2 #19818)
- Known bugs fixed
- 422 or 400 for native histogram disabled (Remote Write 2.0 returns 500 when Native Histograms are sent but the feature is disabled #17181)
- Remote Write V2 and converted classic histograms to NHCB #17075
- Make exemplar implementation follow the spec feat(sending RW2): attach exemplars to samples/histograms in stateless-* modes #17857
- Alternatively consider spec addition (adding Exemplar to "Sample")
- STs are working with RW receiving (storage/remote: migrate Remote Write 2.x receiver to AppenderV2 #19251) (could be descoped?)
Adoption / Documentation / Integrations
- Finalize RW2 sender compliance tests (Overhaul
remotewrite/sendertests compliance#208) - Ship the blog post on RW stability and compliance. ([Remote Write] Write blog post about new compliance tests and library advancements. #17603, Remote Write 2.0: Write a blog post (when stable) #14254)
- Gather latest/contribute/help with PRW 2.0 in Thanos cc @saswatamcode
- Gather latest/contribute/help with PRW 2.0 in Cortex cc @alanprot
- Gather latest/contribute/help with PRW 2.0 in Mimir cc @krajorama
- Gather latest adoption state from vendors/providers (I'm heard some rumours about Google, Grafana Cloud and NewRelic; to double check).
Out of scope for GA / nice-to-have
- Address TODO's in Remote Write Prometheus code and tests (remote write 2.0 - ensure unit tests are all properly updated #13479).
- Finish proposal for reasoning documentation (remote write 2.0 - formal proposal #13887) @bwplotka
- Metadata in WAL related tickets, likely closable, especially less relevant If we assume
metadata-wal-recordsfeature discouraged (IMO)- Metadata vs max samples limit (remote write 2.0 - ensure the metadata is not counted against max samples in a write request #13480).
- metadata + cache (remote write 2.0 - ensure metadata cache in queue manager is cleaned up #13482)
- Consider new compression (remote write 2.0 - implement additional compressions #13366)
- DRY code (remote write 2.0 - refactor append functions, build write request, etc. to reduce code duplication #13481, remote write 2.0 - DRY the queue manager code #14409).
- Prometheus uses the official RW client @saswatamcode on sender.
- This has been attempted, but the special handling for age limited retries and lack of sent bytes metric is blocking. Something to consider adding to lib to have lib used in Prometheus, but not a blocker for stability. [RW] Use remoteapi.Write function to sendSamplesWithBackoff #17373
- Official benchmarks numbers for the protocol itself (Remote Write 2.0: Benchmarks vs 1.0 and OTLP #14253) (FYI prombench already benchmark implementation).
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
GiedriusS commentedon Jul 29, 2025
Thanks for the work on RW2.0. I have a very small topic/issue or maybe even my misunderstanding:
How could those headers help with partial write handling if it is unclear from them which samples were written? If I am sending series A, B, C, and B & C were written successfully, I'd like the headers to tell me that so that I could only retry sending A. At this point in time, the headers will only tell me that 2 series were written successfully out of 3. Could you explain that a little bit?
bwplotka commentedon Jul 29, 2025
Yes and great question @GiedriusS!
They help without that low level information. We started explaining why in our proposal.-,%23%23%23%20Partial%20Writes,-PRW%202.0%20generally), but it could be expanded, I will make note to extend this point.
TL;DR: It's already a huge improvement to know and decide what you want to do with the knowledge IF you are hitting partial write and in what capacity (how many samples) -- on both caller and receiver. It's not trivial (complexity, efficiency) to now communicate back what exactly was not sent and why. Imagine now doubling the network cost and writing/receiving on failures, risking cascading failures. Instead we propose that servers, on partial response, ask for retry with all samples, on fails with the partial "retryable" errors. The server needs to be idempotent as described, even if it's a duplication failure for other samples in the batch (it's acceptable, but one could consider making duplicate sample a 200 e.g. in Thanos).
Does this help?
71 remaining items
prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start …
prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start …
prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start …
bwplotka commentedon Mar 4, 2026
Amazing! Thanks for the recent feedback.
To me (feel free to object) the main points that are needed to make RW2 stable are:
remotewrite/sendertests compliance#208Re some other discussions here:
bwplotka commentedon Mar 4, 2026
Let's unpack the response headers for "async decoding" receivers problem a little.
The intention of response headers was to:
Re (2) it's obvious it's not possible for all implementations. In fact it's likely not possible in most. Like imagine even Otel Collector - it receives and it's unlikely to expect RW2 receiver implementation to wait until all possible export paths returns 200 on every sample. That's exactly why we didn't define precisely what the "successful write is"
In practice it's for protocol communication reasons, so IMO it's totally ok if it the "write" mean only a succesful decoding of the proto, so (2) on @pracucci comment in this discussion:
So the first improvement I propose is to clarify fact in the spec, so clients don't assume it means fully transactional write.
The bigger problem occurs when even the decoding is async as in NewRelic
.
N/Aor-1is something we can add but I see 2 arguments against it:My first instinct is that this way of receiving has quite many problems (debugging wise, clients learning about problems after too long time). It's like UDP vs TCP almost 🙃 -- but should RW protocol block businesses and maybe interesting new use cases by requiring basic decoding to happen before you return success? What if blocking this use case is beneficial because it forces receivers to ensure that basic communication reliance?
As @roidelapluie noticed N/A/-1 path has a risk of receivers ignoring this feature completely for lazy reasons. But again, should protocol guard clients against this forcefully?
Given the above @rthorn-nr how hard (e.g. in score from 0 to 10) would it be for NewRelic to actually decode proto before responding with 200, given strong benefits to clients I mentioned above? Another solution is to add -1/N/A support to Prometheus, but don't mention it in the spec (common implementations being relaxed on this). We could also create a quick meeting to discuss this @roidelapluie @rthorn-nr WDYT?
VihasMakwana commentedon Mar 24, 2026
@bwplotka I was reviewing the list of open issues and was wondering if you’re open to external contributions. I’d be glad to help move things forward where possible.
bwplotka commentedon Mar 24, 2026
Always! This list here should be up to date
rthorn-nr commentedon Mar 27, 2026
Hey @bwplotka, apologies for the delay getting back to you on this. Unfortunately it'd be a 10/10 impossibility to decode the proto before sending a response due to architectural constraints at our edge from the volume of telemetry data we ingest.
To me, this observation begs the question of why this requirement falls under MUST instead of SHOULD/MAY... giving the client the best possible statistics for accounting purposes sounds reasonable, especially for the partial write scenario, but given that it's not possible for all implementations, it sounds like something that arguably should be strongly encouraged rather than required.
If this could indeed be downgraded to SHOULD/MAY, formal support could be added for async backends. Pardon my ignorance of the project, but how much of a priority is it for clients to receive the exact number of datapoints written, even with the loose definition of written that you mentioned? Does this substantially break the retry logic and/or other parts of the proposed spec?
That apt analogy really highlights the difference between synchronous and asynchronous backends. New Relic surfaces issues to customers through error reporting from within our telemetry pipeline, so clients sending telemetry data see only whether their request is successful (i.e. authorization is correct and request is accepted), and everything else (validation, normalization, pruning, etc) happens asynchronously.
So in practice, we'd likely be receiving the request from the sender and returning 200 if authorization is successful, then if we find the proto is invalid when it is decoded, we would emit an error to the customer's account which would be a call to action to fix it on their end. I wouldn't be surprised if other large vendors in the field use a similar async model.
Since there are benefits and tradeoffs to sync and async approaches, would you consider supporting different modes of operation? I think an optional mode of async support would give implementers some options.
rthorn-nr commentedon Jun 8, 2026
Hey @bwplotka 👋, any updates on async backend support?
sriharshabm commentedon Jul 1, 2026
Hi all,
We're currently using Remote Write v1. After reviewing the documentation and tickets related to Remote Write v2, it's clear that a number of performance improvements have been made on top of v1, and we're keen to migrate to v2.
Since Remote Write v2 is still marked as experimental, could you please let us know the expected timeline for it reaching GA (General Availability)?
bwplotka commentedon Aug 4, 2026
While I think it would open room to ignoring validation where it shouldn't I think we need to enable flexibility to do fully async setups, it's getting more popular and discussed. Help wanted to adjust the protocol with this breaking change -- we need clients to support "N/A" in response headers or something similar.
bwplotka commentedon Aug 4, 2026
I will update this ticket with what's the latest, but this async support might be the most important change before GA.
bwplotka commentedon Aug 5, 2026
FYI: We have now weekly syncs to progress on this work
RW 2.0-rc.5: allow exemplar-only time series (#3085)
rw2: rc.5 version (#3080)