[Elasticsearch]: Authentication fails when password contains YAML special characters (e.g. double quotes)

Integration Name

Elasticsearch [elasticsearch]

Dataset Name

All metrics datasets (cluster_stats, node_stats, node, index, index_recovery, index_summary, ccr, enrich, ingest_pipeline, ml_job, pending_tasks, shard)

Integration Version

1.20.1

Agent Version

8.19.1

Agent Output Type

elasticsearch

Elasticsearch Version

8.19.1

OS Version and Architecture

Linux (x86_64)

Software/API Version

N/A

Error Message

The elasticsearch/metrics input enters a Degraded state with HTTP 401 errors when the configured password contains YAML special characters such as double quotes ("):

Degraded

Error fetching data for metricset elasticsearch.cluster_stats: error determining if
connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized

The same error pattern appears across all metrics datasets that share the same template structure.

Event Original

N/A — this is an authentication/configuration issue, not a pipeline/parsing issue.

What did you do?

Configured the Elasticsearch integration's metrics input (elasticsearch/metrics) with a username and password where the password value contains double-quote characters, for example:

password: "mysecret"

(i.e. the literal password string includes the " characters as part of the value, not as YAML delimiters)

What did you see?

All metrics datasets immediately enter a Degraded state with 401 Unauthorized responses from Elasticsearch.

Inspecting the compiled agent configuration (components-actual.yaml) shows the password is stored without the surrounding quote characters:

# With a password value of: "mysecret"
password:
  kind:
    stringvalue: mysecret   # <-- double quotes stripped by YAML parser

This happens because the Handlebars template renders the password directly as an unquoted YAML scalar:

# stream.yml.hbs (all metrics datasets)
{{#if password}}
password: {{password}}
{{/if}}

When {{password}} expands to "mysecret", the rendered YAML line becomes:

password: "mysecret"

YAML interprets the double quotes as string delimiters, so the parsed value is mysecret — the quotes are silently dropped. The agent then authenticates with the wrong password.

Verification with curl confirms the distinction:

# Fails (quotes stripped by shell — same as what the agent experiences):
curl -k -u monitoring_user:"mysecret" -X GET 'https://localhost:9200/_cluster/stats?pretty'
# {"type":"security_exception","reason":"unable to authenticate user..."}

# Succeeds (quotes preserved via single-quoting):
curl -k -u 'monitoring_user:"mysecret"' -X GET 'https://localhost:9200/_cluster/stats?pretty'

Logging in via Kibana UI with the same password works correctly because the UI handles the value as a raw string.

What did you expect to see?

Authentication should succeed regardless of whether the password contains YAML special characters. The compiled agent config should preserve the password value exactly as entered, including any double quotes, backslashes, or other special characters.

Anything else?

Root cause: The .hbs stream templates for every Elasticsearch metrics dataset use password: {{password}} without quoting, so the YAML parser interprets characters like " as structural syntax rather than literal content.

Fix: Replace {{password}} with {{escape_string password}} in all 12 metrics dataset stream templates. The escape_string Handlebars helper (registered in Fleet's template engine) wraps the value in YAML single quotes and escapes any interior single quotes by doubling them, safely preserving all special characters.

For example, a password of "mysecret" would render as:

password: '"mysecret"'

which YAML correctly parses back to the string "mysecret" (with the double quotes intact).

The same pattern is already used correctly in other integrations (e.g. ibmmq, apache_tomcat, websphere_application_server) and with other sensitive fields in cloudflare (escape_string auth_key).

Workaround: Use a password that does not contain YAML special characters, or authenticate via API key instead of username/password.

[Elasticsearch] Fix auth failure when password contains YAML special …

tl;dr: The failure is reproducible and rooted in unquoted credential templating in Elasticsearch stream templates (password: {{password}}), and the direct fix is to switch to escaped/quoted rendering across all 12 Elasticsearch metrics datasets (and preferably username/api_key too for consistency and safety).

Recommendation
Adopt the escaping approach from PR #18435 (or equivalent): replace raw credential interpolation in Elasticsearch stream templates with YAML-safe escaped output (e.g. {{escape_string password}}). Apply the same pattern consistently to username and api_key fields to avoid adjacent YAML scalar edge cases.

Findings

Current Elasticsearch templates render credentials as raw YAML scalars (no quoting/escaping):
- packages/elasticsearch/data_stream/node/agent/stream/stream.yml.hbs:8 → username: {{username}}
- packages/elasticsearch/data_stream/node/agent/stream/stream.yml.hbs:11 → password: {{password}}
- packages/elasticsearch/data_stream/node/agent/stream/stream.yml.hbs:14 → api_key: {{api_key}}
Same password: {{password}} pattern appears in all 12 metrics streams:
- .../cluster_stats/.../stream.yml.hbs:11
- .../node_stats/.../stream.yml.hbs:11
- .../node/.../stream.yml.hbs:11
- .../index/.../stream.yml.hbs:11
- .../index_recovery/.../stream.yml.hbs:12
- .../index_summary/.../stream.yml.hbs:11
- .../ccr/.../stream.yml.hbs:11
- .../enrich/.../stream.yml.hbs:11
- .../ingest_pipeline/.../stream.yml.hbs:11
- .../ml_job/.../stream.yml.hbs:11
- .../pending_tasks/.../stream.yml.hbs:11
- .../shard/.../stream.yml.hbs:11
Manifest confirms these are user-configurable credentials:
- packages/elasticsearch/manifest.yml:46-66 (username, password, api_key).
Repo precedent exists for quoted password rendering:
- packages/apache_tomcat/data_stream/cache/agent/stream/stream.yml.hbs:13 → password: "{{password}}"
- packages/ibmmq/data_stream/qmgr/agent/stream/stream.yml.hbs:14 → password: "{{password}}"
Related PR context:
- #18435 documents this exact issue and proposes {{escape_string password}} for the same 12 templates, but the PR is currently closed and unmerged.
Issue body access limitation:
- I could not read #18434 body text directly via MCP due integrity filtering, so analysis is based on title + code + PR context.

Verification

I verified YAML behavior with a local parser using representative rendered outputs:

$ python - <<'PY'
import yaml
samples=['"abc','abc"','"abc"','"a"b"','"']
for s in samples:
    doc=f'password: {s}\n'
    try:
        v=yaml.safe_load(doc)
        print('OK',repr(s),'=>',v)
    except Exception as e:
        print('ERR',repr(s),type(e).__name__,str(e).splitlines()[0])
PY
ERR '"abc' ScannerError while scanning a quoted scalar
OK 'abc"' => {'password': 'abc"'}
OK '"abc"' => {'password': 'abc'}
ERR '"a"b"' ParserError while parsing a block mapping
ERR '"' ScannerError while scanning a quoted scalar

Additional parser checks also showed plain-scalar pitfalls for special characters (e.g. alias/object coercion):

OK '{x}' => {'password': {'x': None}}
ERR '*alias' ComposerError found undefined alias 'alias'
OK '[a,b]' => {'password': ['a', 'b']}

This confirms raw scalar insertion can change value semantics or produce parse errors depending on input.

Detailed Action Plan

Update Elasticsearch stream templates to YAML-safe credential rendering:
- Files listed above under Findings (12 files).
- Replace raw {{password}} with escaped/quoted form (as in #18435, {{escape_string password}}).
Extend same safety to adjacent credentials:
- In same templates, review username and api_key lines (:8/:14 in most files) and apply equivalent escaping strategy for consistency.
Add regression coverage in Elasticsearch package tests:
- Add auth config cases with password containing ", :, #, ', *, {}, and verify rendered config preserves intended literal value.
- Include at least one api_key case (currently test defaults mostly use simple password: changeme).
Validate package behavior end-to-end:
- Render policy and confirm generated YAML retains full credential literal.
- Run existing Elasticsearch package tests after updates.

Related Items

Type	Link	Relevance
Issue	#18434	Triage target
PR	#18435	Direct fix attempt for this issue (closed/unmerged)
PR	#7637	Prior auth-related change (`api_key`) in Elasticsearch package
File	`packages/elasticsearch/manifest.yml:46-66`	Credential vars definition
File	`packages/elasticsearch/data_stream/node/agent/stream/stream.yml.hbs:8-14`	Representative unescaped credential rendering
File	`packages/apache_tomcat/data_stream/cache/agent/stream/stream.yml.hbs:10-13`	Quoted password precedent
File	`packages/ibmmq/data_stream/qmgr/agent/stream/stream.yml.hbs:11-14`	Quoted password precedent

Note

🔒 Integrity filter blocked 34 items

The following items were blocked because they don't meet the GitHub integrity level.

#18434 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#18434 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17816 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#18265 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17817 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17973 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17381 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#15525 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#14985 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#7886 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#16962 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17861 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17644 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17623 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#17189 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#16917 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
... and 18 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[Elasticsearch] Fix authentication failure when password contains YAM…

Package elasticsearch - 1.20.2 containing this change is available at https://epr.elastic.co/package/elasticsearch/1.20.2/

[Elasticsearch] Fix authentication failure when password contains YAM…

Navigation Menu

Repository navigation

[Elasticsearch]: Authentication fails when password contains YAML special characters (e.g. double quotes) #18434

Integration Name

Dataset Name

Integration Version

Agent Version

Agent Output Type

Elasticsearch Version

OS Version and Architecture

Software/API Version

Error Message

Event Original

What did you do?

What did you see?

What did you expect to see?

Anything else?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Elasticsearch]: Authentication fails when password contains YAML special characters (e.g. double quotes) #18434

Description

Integration Name

Dataset Name

Integration Version

Agent Version

Agent Output Type

Elasticsearch Version

OS Version and Architecture

Software/API Version

Error Message

Event Original

What did you do?

What did you see?

What did you expect to see?

Anything else?

Activity

github-actions commented on Apr 16, 2026

elastic-vault-github-plugin-prod commented on May 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions