Repository navigation
Fix uri_parts processor behaviour for missing extensions - #105689
Merged
Merged
Conversation
Collaborator
|
Hi @nielsbauman, I've created a changelog YAML for you. |
Collaborator
|
Pinging @elastic/es-data-management (Team:Data Management) |
joegallo
reviewed
Feb 21, 2024
Contributor
On I'm not actually sure I care about the old behavior, but I don't think we should change it unintentionally. Can you add a test for the old behavior and switch it back? I'd include a comment in the test about how it just happens to be the case that we do this, and it's not necessarily something we intend to keep forever but that if we change it we want to be conscious of it being a change in the observable behavior of the system. |
joegallo
approved these changes
Feb 21, 2024
1 of 4 tasks
andrewkroh
added a commit
to elastic/integrations
that referenced
this pull request
Apr 17, 2024
Ignore the existence of an invalid `url.extension` field. Stack versions < 8.14 had a
bug that populated the field with bad data. After a package uses a minimum stack version
of 8.14.0 then this addition to `dynamic_fields` can be removed.
This fixes errors like this which occur under v8.14.0
test case failed: Expected results are different from actual ones: --- want
+++ got
@@ -1797,7 +1797,6 @@
"preserve_duplicate_custom_fields"
],
"url": {
- "extension": "com/page",
"original": "www.example.com/page",
"path": "www.example.com/page"
}
Relates: elastic/elasticsearch#105689
1 of 4 tasks
kcreddy
added a commit
to elastic/integrations
that referenced
this pull request
Aug 6, 2024
Extension to #9623 Ignore the existence of an invalid url.extension field. Stack versions < 8.14 had a bug that populated the field with bad data. After a package uses a minimum stack version of 8.14.0 then this addition to dynamic_fields can be removed. This fixes errors like this which occur under v8.14.0+ +++ got @@ -1797,7 +1797,6 @@ "preserve_duplicate_custom_fields" ], "url": { - "extension": "com/page", "original": "www.example.com/page", "path": "www.example.com/page" } Relates: elastic/elasticsearch#105689
4 tasks done
3 of 5 tasks
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 4, 2025
Extension to elastic#9623 Ignore the existence of an invalid url.extension field. Stack versions < 8.14 had a bug that populated the field with bad data. After a package uses a minimum stack version of 8.14.0 then this addition to dynamic_fields can be removed. This fixes errors like this which occur under v8.14.0+ +++ got @@ -1797,7 +1797,6 @@ "preserve_duplicate_custom_fields" ], "url": { - "extension": "com/page", "original": "www.example.com/page", "path": "www.example.com/page" } Relates: elastic/elasticsearch#105689
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 5, 2025
Extension to elastic#9623 Ignore the existence of an invalid url.extension field. Stack versions < 8.14 had a bug that populated the field with bad data. After a package uses a minimum stack version of 8.14.0 then this addition to dynamic_fields can be removed. This fixes errors like this which occur under v8.14.0+ +++ got @@ -1797,7 +1797,6 @@ "preserve_duplicate_custom_fields" ], "url": { - "extension": "com/page", "original": "www.example.com/page", "path": "www.example.com/page" } Relates: elastic/elasticsearch#105689
catinspace-au
added a commit
to hyperi-io/dfe-transform-elastic
that referenced
this pull request
Aug 25, 2026
Everything m365_defender and microsoft_defender_endpoint were losing, plus the url extension that went with it. Three of the four are new Painless shapes, all read off the script rather than hard-coded. `process.name` is gathered from the executables of `process.command_line` -- first token, last `/` segment, quotes stripped -- and written as a SCALAR when one name survives and a list otherwise, which is the vendor's own HashSet ending. DNS header flags are the keys of a literal list whose value in the map is the string "true", in the list's order rather than the map's. And DNS answers zip with their TTLs into a list of objects, carrying the script's cast to long and its own error message when the two lists disagree in length. The fourth is the sentinel removal. The defender pipelines wrap it in a recursive drop-empty and run `drop(ctx)` after it, so the removal claimed the script and the sweep that clears what it just emptied never ran -- an evidence list the vendor ships empty came out as a stray `[]`. url.extension comes from the segment after the LAST `/`, and a path with no slash at all gets none. That is elastic/elasticsearch#105689, which we had half of: the dotted directory was already handled, the bare hostname was not, so `subdomain.domain.tld` came out with an extension of "tld". zscaler_zia 92 -> 93 of 93, m365_defender 100 -> 106, defender endpoint 20 -> 22, no source regressed. Corpus 4,807/4,850 events (99%) from 4,798, fields 100.0%. Baseline raised, suite green, clippy clean.
catinspace-au
added a commit
to hyperi-io/dfe-transform-elastic
that referenced
this pull request
Sep 4, 2026
Everything m365_defender and microsoft_defender_endpoint were losing, plus the url extension that went with it. Three of the four are new Painless shapes, all read off the script rather than hard-coded. `process.name` is gathered from the executables of `process.command_line` -- first token, last `/` segment, quotes stripped -- and written as a SCALAR when one name survives and a list otherwise, which is the vendor's own HashSet ending. DNS header flags are the keys of a literal list whose value in the map is the string "true", in the list's order rather than the map's. And DNS answers zip with their TTLs into a list of objects, carrying the script's cast to long and its own error message when the two lists disagree in length. The fourth is the sentinel removal. The defender pipelines wrap it in a recursive drop-empty and run `drop(ctx)` after it, so the removal claimed the script and the sweep that clears what it just emptied never ran -- an evidence list the vendor ships empty came out as a stray `[]`. url.extension comes from the segment after the LAST `/`, and a path with no slash at all gets none. That is elastic/elasticsearch#105689, which we had half of: the dotted directory was already handled, the bare hostname was not, so `subdomain.domain.tld` came out with an extension of "tld". zscaler_zia 92 -> 93 of 93, m365_defender 100 -> 106, defender endpoint 20 -> 22, no source regressed. Corpus 4,807/4,850 events (99%) from 4,798, fields 100.0%. Baseline raised, suite green, clippy clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
uri_partsprocessor was behaving incorrectly for URI's that included a dot in the path but did not have an extension. For example:https://www.example.com/path.withdot/filenamewithoutextensionwould get processed as"extension": "withdot/filenamewithoutextension".Instead of determining the extension based on the full path, we should use the last segment of the path (i.e. the part after the last
/).This PR also includes (missing) YAML REST tests for the
uri_partsprocessor.Fixes #105612