Error in ES keystore when elasticsearch.yml references custom variables in /etc/default/elasticsearch

[gchiartosini (GIADA CHIARTOSINI)]

Hello,
can anyone help with this weird problem, please?
I have defined custom variables in /etc/default/elasticsearch file, and referenced them in the elasticsearch.yml; ES works with them properly, but I get an error about those variables whenever I try to run any command (list, add) on the elasticsearch keystore.
What am I doing wrong?

I installed the ELK 7.6 (Debian) stack, configured the xpack security and the keystore without errors.

Only after inserting custom variables into the /etc/default/elasticsearch file, and having successfully used them in the elasticsearch.yml file, I am no longer able to use the keystore.

I get the following error:

$ cd /usr/share/elasticsearch
$ sudo bin / elasticsearch-keystore list
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'ESC_SMTP_FROM'
	at org.elasticsearch.common.settings.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:116)
	at org.elasticsearch.common.settings.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:69)
	at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1162)
	at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1118)
	at org.elasticsearch.node.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:108)
	at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:90)
	at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
	at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)

Lines added in the /etc/default/elasticsearch file:

#################################
## TLP NCR CUSTOM VARIABLES
#################################
# For /etc/elasticsearch/elasticsearch.yml
# section: xpack.notification.email.account
ES_SMTP_FROM="noreply@example.com"
ES_SMTP_SSL_TRUST="10.123.123.45"
ES_SMTP_AUTH=false
ES_SMTP_HOST="10.123.123.45"
ES_SMTP_PORT=25
ES_SMTP_USER="noreply@example.com"

Lines added in the elasticsearch.yml file:

# ---------------------------------- SMTP for Watchers -------------------------#
xpack.notification.email.account.elk_account.profile: standard
xpack.notification.email.account.elk_account.email_defaults.from: "${ESC_SMTP_FROM}"
xpack.notification.email.account.elk_account.smtp.ssl.trust: "${ESC_SMTP_SSL_TRUST}"
xpack.notification.email.account.elk_account.smtp.auth: "${ESC_SMTP_AUTH}"
xpack.notification.email.account.elk_account.smtp.host: "${ESC_SMTP_HOST}"
xpack.notification.email.account.elk_account.smtp.port: "${ESC_SMTP_PORT}"
xpack.notification.email.account.elk_account.smtp.user: "${ESC_SMTP_USER}"
xpack.notification.email.default_account: elk_account

Note: Elasticsearch works correctly.
Note2: SMTP configuration works correctly.
Note3: If I remove the variables from the .yml file the keystore no longer returns errors.

Thanks for your help,
Giada

[gchiartosini (GIADA CHIARTOSINI)]

Sorry, I have included old configuration lines for the /etc/default/elasticsearch file.

These are the correct lines:

#################################
## TLP NCR CUSTOM VARIABLES
#################################
# For /etc/elasticsearch/elasticsearch.yml
# section: xpack.notification.email.account
ESC_SMTP_FROM="noreply@telepass.com"
ESC_SMTP_SSL_TRUST="10.153.248.39"
ESC_SMTP_AUTH=false
ESC_SMTP_HOST="10.153.248.39"
ESC_SMTP_PORT=25
ESC_SMTP_USER="noreply@telepass.com"

[ikakavas (Ioannis Kakavas)]

/etc/default/elasticsearch is not an alternative configuration file, it only allows you to set some parameters. You can not add arbitrary substitutions in there.

What you are actually trying to use is environment variable substituion, but as the name alludes to, you would have to export ESC_SMTP_FROM etc as environment variables.

The reason why it works for elasticsearch, is that elasticsearch runs as a service and the service definition loads and sources /etc/default/elasticsearch thus making anything you define in there available as an environment variable.

[gchiartosini (GIADA CHIARTOSINI)]

Thank you very much for your clarification about the function of the /etc/default/elasticsearch file and the loading process.

The weird thing is that the elasticsearch-keystore utility uses /bin/elasticsearch-env which in turn does source /etc/default/elasticsearch - apparently.

Our aim would be to store our custom environment variables in one place where both elasticsearch as a service, and the elasticsearch-keystore utility from command line via sudo, could read them.

Is that at all possible, that you know?

We have even tried to move our custom variables to system-wide file

• /etc/environment

but it was worse, because elasticsearch was not able to resolve them.