Watcher: Support for Keystore Variables in Watch Definitions

console_fulcrum
Wed, 01 Oct 2025 18:41:05 +0000
Elasticsearch

Currently working on Elasticsearch 8.17.1 with Watcher + DataDog integration. While xpack.watcher.encrypt_sensitive_data=true encrypts stored watches, the initial watch definition still requires plaintext credentials.

Current Limitation

{
  "actions": {
    "datadog_webhook": {
      "webhook": {
        "headers": {
          "DD-API-KEY": "plaintext-api-key-here",
          "DD-APPLICATION-KEY": "plaintext-app-key-here"
        }
      }
    }
  }

This creates issues for:

Source control: Watches stored in Git contain plaintext credentials

CI/CD pipelines: Automated deployments expose secrets

Team collaboration: Sharing watch definitions requires credential management

Attempted Solution

Tried using keystore variables (similar to elasticsearch.yml):

bin/elasticsearch-keystore add DD_API_KEY

"DD-API-KEY": "{{_secrets.DD_API_KEY}}"

Result

unknown secure setting [DD_API_KEY]` - keystore only accepts predefined settings.

Where DD_API_KEY is stored securely in the keystore, similar to how elasticsearch.yml supports ${keystore.setting} syntax.

Use Cases

DataDog, PagerDuty, Slack webhook authentication

SMTP credentials for email actions

Custom webhook APIs requiring authentication

Any third-party service integration

Current Workarounds

1. Encryption at rest: xpack.watcher.encrypt_sensitive_data=true (but plaintext in source)

2. External templating: Pre-process watch definitions before deployment

3. Environment variables: Limited Docker/container solutions

Would keystore variable support be feasible for future releases? This would align with Elasticsearch's existing keystore patterns and solve a common security concern for teams managing watches in source control.

6 posts - 2 participants