# Watcher: Support for Keystore Variables in Watch Definitions

**URL:** https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting, docker
**Created:** [October 1, 2025, 6:41pm UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362 "2025-10-01T18:41:05Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![console\_fulcrum](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/console_fulcrum/32/142169_2.png) [@console\_fulcrum](https://discuss.elastic.co/u/console_fulcrum)
#### Post date: [October 1, 2025, 6:41pm UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362/1 "2025-10-01T18:41:05Z")

</div>

Currently working on Elasticsearch 8.17.1 with Watcher + DataDog integration. While **`xpack.watcher.encrypt_sensitive_data=true`** encrypts stored watches, **the initial watch definition still requires plaintext credentials**.

## Current Limitation

```auto
{
  "actions": {
    "datadog_webhook": {
      "webhook": {
        "headers": {
          "DD-API-KEY": "plaintext-api-key-here",
          "DD-APPLICATION-KEY": "plaintext-app-key-here"
        }
      }
    }
  }

```

This creates issues for:

- **Source control** : Watches stored in Git contain plaintext credentials
- **CI/CD pipelines** : Automated deployments expose secrets
- **Team collaboration** : Sharing watch definitions requires credential management

**Attempted Solution**

Tried using keystore variables (similar to elasticsearch.yml):

```auto
bin/elasticsearch-keystore add DD_API_KEY

"DD-API-KEY": "{{_secrets.DD_API_KEY}}"

```

**Result**

```auto
unknown secure setting [DD_API_KEY]` - keystore only accepts predefined settings.

```

Where **`DD_API_KEY`** is stored securely in the keystore, similar to how **`elasticsearch.yml`** supports **`${keystore.setting}`** syntax.

**Use Cases**

- DataDog, PagerDuty, Slack webhook authentication
- SMTP credentials for email actions
- Custom webhook APIs requiring authentication
- Any third-party service integration

**Current Workarounds**

1. **Encryption at rest** : **`xpack.watcher.encrypt_sensitive_data=true`** (but plaintext in source)
2. **External templating** : Pre-process watch definitions before deployment
3. **Environment variables** : Limited Docker/container solutions

Would keystore variable support be feasible for future releases? This would align with Elasticsearch's existing keystore patterns and solve a common security concern for teams managing watches in source control.

---

<div class="post-metadata">

### Author: ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)
#### Post date: [October 2, 2025, 3:40pm UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362/2 "2025-10-02T15:40:04Z")

</div>

This does sound like it could be an interesting feature. Would you mind opening a public Github issue on the Elasticsearch repository with this information?

---

<div class="post-metadata">

### Author: ![console\_fulcrum](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/console_fulcrum/32/142169_2.png) [@console\_fulcrum](https://discuss.elastic.co/u/console_fulcrum)
#### Post date: [October 3, 2025, 4:52am UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362/3 "2025-10-03T04:52:05Z")

</div>

Hi @dakrone , sure will do.

On the side, could you help me by confirming that

- We cannot as of today use the Elasticsearch Keystore - to store custom secrets, so it can be viewed as encrypted in the Watcher UI as well?  
Thank you.

---

<div class="post-metadata">

### Author: ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)
#### Post date: [October 3, 2025, 5:51pm UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362/4 "2025-10-03T17:51:23Z")

</div>

> [@console\_fulcrum](#):
>
> We cannot as of today use the Elasticsearch Keystore - to store custom secrets, so it can be viewed as encrypted in the Watcher UI as well?

I believe you are correct that you cannot view these values from the Keystore in the Watcher UI today.

---

<div class="post-metadata">

### Author: ![console\_fulcrum](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/console_fulcrum/32/142169_2.png) [@console\_fulcrum](https://discuss.elastic.co/u/console_fulcrum)
#### Post date: [October 5, 2025, 3:03pm UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362/5 "2025-10-05T15:03:50Z")

</div>

Hi @dakrone , I have raised the issue here. - [[XPack][Watcher]: Support for Keystore Variables in Watch Definitions for Secure Secret Management · Issue #136001 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/136001)

---

<div class="post-metadata">

### Author: ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)
#### Post date: [October 6, 2025, 2:33pm UTC](https://discuss.elastic.co/t/watcher-support-for-keystore-variables-in-watch-definitions/382362/6 "2025-10-06T14:33:53Z")

</div>

Thanks! I’ve triaged it to the appropriate team.
