Why doesn't add_tag => ["${HOSTNAME}"] work?

Elastic Stack Logstash
1

This is an environment variable, but I can't get it to work? The tag save is, ${HOSTNAME}, not the environment variable value.

https://www.elastic.co/guide/en/logstash/current/environment-variables.html#_setting_the_value_of_a_tag

filter {  
  mutate {
    add_tag => ["${HOSTNAME}"]
  }
...
}
2

How do you know the HOSTNAME variable is set? Have you inspected /proc/$LOGSTASH_PID/environ?

3

I am trying to use the environment variable that I can see with printenv.

4

Okay, but I'd still check what environment variables the actual Logstash process is seeing. Also, which version of Logstash is this?

5

v6.2, and I don't see it in /proc/$LOGSTASH_PID/environ.

6

I'm really confused because this is used in the elasticsearch.yml, but doesn't work in logstash?

node.name: ${HOSTNAME}
7

elasticsearch.yml and Logstash pipeline configuration files are entirely different beast, but ${HOSTNAME} should work if that environment variable is set, but it seems like it isn't. Are you starting Logstash from your shell or via an init script (or the systemd equivalent)?

8

systemd:

systemctl start logstash

9

Okay, but the environment variables are set completely differently in the systemd case. According to https://superuser.com/questions/132489/hostname-environment-variable-on-linux $HOSTNAME is a bash invention.

10

So, I looked in the elasticsearch-env file and saw, export HOSTNAME=$HOSTNAME, and I added it to logstash.lib.sh. It now shows in /proc/$LOGSTASH_PID/environ correctly. But now logstash logs an error!

[root@LMUWU0438 bin]# cat /proc/5254/environ

HOSTNAME=LMUWU0438GEM_HOME=/usr/share/logstash/vendor/bundle/jruby/2.3.0SHELL=/sbin/nologinLS_GROUP=logstashLS_HOME=/usr/share/logstashLS_NICE=19LS_JVM_OPTS=/etc/logstash/jvm.optionsJAVA_OPTS=-Xms256m -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError  USER=logstashSERVICE_DESCRIPTION=logstashPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binLS_OPEN_FILES=16384PWD=/LS_SETTINGS_DIR=/etc/logstashLS_PIDFILE=/var/run/logstash.pidLANG=en_US.UTF-8SERVICE_NAME=logstashLS_USER=logstashSHLVL=0HOME=/opt/logstashLOGNAME=logstashLS_GC_LOG_FILE=/var/log/logstash/gc.logGEM_PATH=/usr/share/logstash/vendor/bundle/jruby/2.3.0JAVACMD=/bin/javaSINCEDB_DIR=/usr/share/logstashLOGSTASH_HOME=/usr/share/logstash
11

But now logstash logs an error!

Namely....?

12

It's too big to paste here. Not sure how to post it here?

13

Use pastebin or a GitHub gist.

14

I set logging to debug. (I fixed the link)

15

[2018-04-24T16:04:50,977][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::AccessException", :message=>"Could not determine keystore password. Please ensure the file at /etc/logstash/logstash.keystore is a valid Logstash keystore",
...

It's not obvious why this would have anything to do with the addition of the HOSTNAME variable. Are you sure it only happens when you set it in logstash.lib.sh?

16

No, it only happens when I have the mutate entry. I'm going to start logstash with debug logging without the mutate and put the log up.

17

Mutate is commented out. Runs fine.

18

@magnusbaeck, FYI, when I run this pipeline locally, I get a different error!

I have created a support ticket, and once resolved will update here. Thank you for all of your help!

19

@magnusbaeck, With the help of elastic support, we figured this out. Turns out it was 2 easy changes to get it working. I am not saying that it was easy figuring this out, because it wasn't easy!

mutate {
  add_tag => ["${HOSTNAME}"]
}
  • Add xpack.management.pipeline.id: to logstash.yml
  • Create file, /etc/sysconfig/logstash, (chmod 600) and add these 2 lines
    • HOSTNAME=ThisHostName
    • LOGSTASH_KEYSTORE_PASS=password

Thanks again for your help!

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.