Currently working on Elasticsearch 8.17.1 with Watcher + DataDog integration. While xpack.watcher.encrypt_sensitive_data=true
encrypts stored watches, the initial watch definition still requires plaintext credentials.
Current Limitation
{
"actions": {
"datadog_webhook": {
"webhook": {
"headers": {
"DD-API-KEY": "plaintext-api-key-here",
"DD-APPLICATION-KEY": "plaintext-app-key-here"
}
}
}
}
This creates issues for:
- Source control: Watches stored in Git contain plaintext credentials
- CI/CD pipelines: Automated deployments expose secrets
- Team collaboration: Sharing watch definitions requires credential management
Attempted Solution
Tried using keystore variables (similar to elasticsearch.yml):
bin/elasticsearch-keystore add DD_API_KEY
"DD-API-KEY": "{{_secrets.DD_API_KEY}}"
Result
unknown secure setting [DD_API_KEY]` - keystore only accepts predefined settings.
Where DD_API_KEY
is stored securely in the keystore, similar to how elasticsearch.yml
supports ${keystore.setting}
syntax.
Use Cases
- DataDog, PagerDuty, Slack webhook authentication
- SMTP credentials for email actions
- Custom webhook APIs requiring authentication
- Any third-party service integration
Current Workarounds
- Encryption at rest:
xpack.watcher.encrypt_sensitive_data=true
(but plaintext in source) - External templating: Pre-process watch definitions before deployment
- Environment variables: Limited Docker/container solutions
Would keystore variable support be feasible for future releases? This would align with Elasticsearch's existing keystore patterns and solve a common security concern for teams managing watches in source control.