Watcher: Support for Keystore Variables in Watch Definitions

Elastic Stack Elasticsearch
,
1

Currently working on Elasticsearch 8.17.1 with Watcher + DataDog integration. While xpack.watcher.encrypt_sensitive_data=true encrypts stored watches, the initial watch definition still requires plaintext credentials.

Current Limitation

{
  "actions": {
    "datadog_webhook": {
      "webhook": {
        "headers": {
          "DD-API-KEY": "plaintext-api-key-here",
          "DD-APPLICATION-KEY": "plaintext-app-key-here"
        }
      }
    }
  }

This creates issues for:

  • Source control: Watches stored in Git contain plaintext credentials
  • CI/CD pipelines: Automated deployments expose secrets
  • Team collaboration: Sharing watch definitions requires credential management

Attempted Solution

Tried using keystore variables (similar to elasticsearch.yml):

bin/elasticsearch-keystore add DD_API_KEY

"DD-API-KEY": "{{_secrets.DD_API_KEY}}"

Result

unknown secure setting [DD_API_KEY]` - keystore only accepts predefined settings.

Where DD_API_KEY is stored securely in the keystore, similar to how elasticsearch.yml supports ${keystore.setting} syntax.

Use Cases

  • DataDog, PagerDuty, Slack webhook authentication
  • SMTP credentials for email actions
  • Custom webhook APIs requiring authentication
  • Any third-party service integration

Current Workarounds

  1. Encryption at rest: xpack.watcher.encrypt_sensitive_data=true (but plaintext in source)
  2. External templating: Pre-process watch definitions before deployment
  3. Environment variables: Limited Docker/container solutions

Would keystore variable support be feasible for future releases? This would align with Elasticsearch's existing keystore patterns and solve a common security concern for teams managing watches in source control.

2

This does sound like it could be an interesting feature. Would you mind opening a public Github issue on the Elasticsearch repository with this information?

3

Hi @dakrone , sure will do.

On the side, could you help me by confirming that

  • We cannot as of today use the Elasticsearch Keystore - to store custom secrets, so it can be viewed as encrypted in the Watcher UI as well?
    Thank you.
4

I believe you are correct that you cannot view these values from the Keystore in the Watcher UI today.

5

Hi @dakrone , I have raised the issue here. - [XPack][Watcher]: Support for Keystore Variables in Watch Definitions for Secure Secret Management · Issue #136001 · elastic/elasticsearch · GitHub

6

Thanks! I’ve triaged it to the appropriate team.