Add tie breaker field mapping by Mallets · Pull Request #6859 · quickwit-oss/quickwit

Mallets · 2026-10-01T13:27:35Z

Description

Adds a new tie_breaker field mapping type, backed by Tantivy's generated TieBreaker fast field introduced in quickwit-oss/tantivy#3143.

A tie-breaker field is a u64 fast field whose values are generated at indexing time rather than read from documents. Within a split, values are consecutive starting from a random offset, and they are preserved when splits are merged, so a document keeps its value for its whole lifetime. The intended use is as a secondary sort key that reduces ties among documents with equal primary sort values.

Values are almost always distinct across splits but not guaranteed unique (they are currently limited to the u32 range). Paginating with search_after on a primary sort field plus a tie_breaker field can therefore skip documents that share both values; _shard_doc remains the way to get exact pagination.

Example doc mapping:

field_mappings:
  - name: tie_breaker
    type: tie_breaker

Behavior

Mapping: the type takes no parameters; any parameter is rejected. It serializes as {"name": ..., "type": "tie_breaker"}. array<tie_breaker> is not supported.
Doc mapping updates: adding a tie_breaker field (including nested in an object) through a doc mapping update is rejected, because existing splits would have no values for it. The index must be created with the field.
Indexing: values supplied in documents for this field are ignored.
Doc retrieval: the field is not stored, so it does not appear in _source/hits; it is readable as a fast field.
Queries:
- Term / full-text queries on the field return a schema error (tie_breaker fields are not term-searchable).
- Range queries are supported, treated as a u64 fast-field range.
Elasticsearch _mapping API: reported as long, like u64.
Docs: documented in docs/configuration/index-config.md.

Other changes

Pin tantivy / tantivy-common to 4abc20f, the merge commit of feat: add generated tie-breaker fast fields tantivy#3143 (tip of Tantivy main).
Adapt to the new AggContextParams::value_sources field by constructing it with AggContextParams::new(...).

How was this PR tested?

New unit tests:
- tie_breaker mapping round-trips through serialization/deserialization.
- tie_breaker mappings with parameters are rejected.
- tie_breaker type id parses and serializes correctly.
- Values supplied in a document for a tie_breaker field are ignored.
- Doc mapping updates adding a tie_breaker field are rejected.
cargo check on the full workspace; tests of quickwit-doc-mapper, quickwit-config and quickwit-query.
make fmt.

Not yet covered by tests: sorting on a tie_breaker field, the term-query error, and the Elasticsearch _mapping type.

chatgpt-codex-connector · 2026-10-02T08:23:43Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-02T16:53:06.817740Z	`28e1f52`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0c128488b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…pdates

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2a545cc4d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81fab0eff2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…validation

trinity-1686a · 2026-10-02T15:02:40Z

+
+The field has no parameters. Values provided in documents are ignored, and generated values are preserved across merges. Values are almost always distinct but not guaranteed to be unique, so they should not be used as document identifiers. In particular, paginating with `search_after` on a primary sort field and a `tie_breaker` field can skip documents that share both values. Sort on `_shard_doc` instead when every document must be returned exactly once.
+
+The field is not stored and only supports range queries. `array<tie_breaker>` is not supported, and a `tie_breaker` field cannot be added to an existing index by updating its doc mapping: the index must be created with it.


so it's fast but not stored and not indexed? (to use the terminology of normal fields)

i don't recall wether we allow term:value to match on fast non-indexed values by transforming into a term:[value TO value] query, or if we require the caller to do that themself

Yes. It's fast, not stored and not indexed. The docs say it behaves like a u64 field with fast: true, stored: false and indexed: false, and that term and range queries run on the fast field.

Quickwit does the conversion for fast-only fields: it builds a regular term and Tantivy falls back to an exact range on the fast field when it's not indexed.

d8f657a builds the same u64 term as a fast-only u64 field (see here). A tie_breaker:123 query now works.

trinity-1686a · 2026-10-02T15:04:55Z

+type: tie_breaker
+```
+
+The field has no parameters. Values provided in documents are ignored, and generated values are preserved across merges. Values are almost always distinct but not guaranteed to be unique, so they should not be used as document identifiers. In particular, paginating with `search_after` on a primary sort field and a `tie_breaker` field can skip documents that share both values. Sort on `_shard_doc` instead when every document must be returned exactly once.


i think in strict doc mapping this should reject the document, in lenient and dynamic ignored is correct

(also, because _shard_doc is mentioned, worth saying that contrary to tiebreakers, it isn't stable in quickwit)

0b987ba rejects documents that provide a tie_breaker value in strict_mode.

I removed the mention of _shard_doc from the docs in 53afd15.

trinity-1686a · 2026-10-02T15:11:50Z

+    // Tie-breaker values are only generated for newly indexed documents, so existing splits would
+    // have no values for a tie-breaker field added by an update.


unless having docs with a value and other without causes specific problems, i don't think this should be enforced: it's fine to add fields in general, they will be missing from previous docs (and for non-generated fields, they can even be missing from new documents)

Makes sense, removed in 10c437e. Docs indexed before the update just won't have a value, same as any newly added field. I updated the docs to mention it.

trinity-1686a · 2026-10-02T15:17:07Z

        LeafType::IpAddr(_) => value_to_ip(value),
        LeafType::F64(numeric_options) => value_to_float(value, numeric_options),
        LeafType::U64(numeric_options) => value_to_u64(value, numeric_options),
+        LeafType::TieBreaker => match value {


i think it would make sense to be more lenient and use value_to_u64 here

Done in 27a9807. The numeric options are built explicitly (fast only, not stored, not indexed, no coercion), with a comment noting that only output_format (Number) is actually read by value_to_u64.

trinity-1686a · 2026-10-02T15:22:01Z

+    };
+
+    let ascending_hits = search_hits(SortOrder::Asc).await;
+    assert!(ascending_hits.windows(2).all(|hits| hits[0].2 <= hits[1].2));


https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.is_sorted_by

Done in 27a9807

… in test

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27a9807dc0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign in to view

+
+The field has no parameters. Values provided in documents are ignored, and generated values are preserved across merges. Values are almost always distinct but not guaranteed to be unique, so they should not be used as document identifiers. In particular, paginating with `search_after` on a primary sort field and a `tie_breaker` field can skip documents that share both values. Sort on `_shard_doc` instead when every document must be returned exactly once.
+
+The field is not stored and only supports range queries. `array<tie_breaker>` is not supported. When a `tie_breaker` field is added to an existing index by updating its doc mapping, documents indexed before the update have no value for it.


Mallets added 6 commits October 1, 2026 15:27

Add tie breaker field mapping

69c1763

Update Tantivy lockfile

d73e4f9

Bump Tantivy to latest mallets/tiebreaker

4fd3028

Update Tantivy lockfile

ca80cbb

Merge branch 'main' into mallets/tiebreaker

f1a973f

Document tie_breaker field type

a0c1284

Mallets marked this pull request as ready for review October 2, 2026 08:23

Mallets requested a review from a team as a code owner October 2, 2026 08:23

chatgpt-codex-connector Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread quickwit/Cargo.toml Outdated

Comment thread quickwit/quickwit-doc-mapper/src/doc_mapper/field_mapping_entry.rs Outdated

Comment thread quickwit/quickwit-doc-mapper/src/doc_mapper/mapping_tree.rs

Comment thread docs/configuration/index-config.md Outdated

Mallets added 3 commits October 2, 2026 11:07

Pin Tantivy to merged tie-breaker commit

6df8277

Reject tie_breaker options and tie_breaker additions on doc mapping u…

99e3f86

…pdates

Clarify tie_breaker uniqueness and update constraints in docs

b2a545c

Mallets self-assigned this Oct 2, 2026

chatgpt-codex-connector Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread quickwit/quickwit-config/src/index_config/mod.rs Outdated

Limit Cargo.lock changes to Tantivy

81fab0e

chatgpt-codex-connector Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread quickwit/quickwit-doc-mapper/src/doc_mapper/mapping_tree.rs

Mallets added 2 commits October 2, 2026 12:03

Escape dotted tie_breaker paths and ignore tie_breaker values during …

5c553d7

…validation

Test tie_breaker sorting, queries and Elasticsearch mapping

87cbe73

trinity-1686a reviewed Oct 2, 2026

View reviewed changes

Mallets added 2 commits October 2, 2026 17:56

Allow adding tie_breaker fields through doc mapping updates

10c437e

Convert tie_breaker values with value_to_u64 and use is_sorted_by_key…

27a9807

… in test

chatgpt-codex-connector Bot reviewed Oct 2, 2026

View reviewed changes

Mallets added 4 commits October 2, 2026 18:08

Support term queries on tie_breaker fields

d8f657a

Reject documents providing tie_breaker values in strict mode

0b987ba

Clarify tie_breaker docs and drop _shard_doc recommendation

53afd15

Merge branch 'main' into mallets/tiebreaker

28e1f52


		The field has no parameters. Values provided in documents are ignored, and generated values are preserved across merges. Values are almost always distinct but not guaranteed to be unique, so they should not be used as document identifiers. In particular, paginating with `search_after` on a primary sort field and a `tie_breaker` field can skip documents that share both values. Sort on `_shard_doc` instead when every document must be returned exactly once.

		The field is not stored and only supports range queries. `array<tie_breaker>` is not supported, and a `tie_breaker` field cannot be added to an existing index by updating its doc mapping: the index must be created with it.

		// Tie-breaker values are only generated for newly indexed documents, so existing splits would
		// have no values for a tie-breaker field added by an update.

Conversation

Mallets commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Behavior

Other changes

How was this PR tested?

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mallets Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

This comment was marked as low quality.

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mallets commented Oct 1, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 2, 2026 •

edited

Loading

Mallets Oct 2, 2026 •

edited

Loading