Filebeat.yml - Environnement variable on UBUNTU

[ASRLO (ROCKS)]

Hello everybody ,

I want to use an environment variable on my path but it doesn't work. When I run the filebeat service it works but fails when sending logs to logstash

I have set my environment variable in /etc/bash.bashrc:
export DATE_D=$(date +%d)

Here is the part of my .yml file where the variable is located:

filebeat.inputs:
- type: log
  paths:
    - /var/log/SYSLOG-NG/$DATE_Y/$DATE_M/$DATE_D/WINDOWS/10.1.251.2/*.log
  fields:
    log_type: windows
  fields_under_root: true
- TYPE: log
  paths:
    - /var/log/SYSLOG-NG/$DATE_Y/$DATE_M/$DATE_D/ANTIVIRUS_ESEP/*/*.log
  fields:
    log_type: esep
  fields_under_root: true

I tried to put with and without the "}" but the it does not work
I would like all the service to start correctly
Can you help me ?

[leandrojmp (Leandro Pereira)]

Use the </> code button when sharing code or config.

It looks that your yml does not have the correct format and you need to use ${} around your variable, I would also recommend to use quotes around your config.

It should be something like this.

- type: log
  enabled: true
  paths:
    - "/var/log/SYSLOG-NG/${DATE_D}/WINDOWS/10.1.251.2/*.log"
  fields:
    log_type: windows
  fields_under_root: true

Try this and look at the filebeat log to see if it is harvesting your files.

[ASRLO (ROCKS)]

Thank you for your reply.

Here is the result when I restart the service:

 ● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
       Loaded: loaded (/lib/systemd/system/filebeat.service; disabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Fri 2020-10-30 09:42:58 CET; 3s ago
         Docs: https://www.elastic.co/products/beats/filebeat
      Process: 6948 ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
     Main PID: 6948 (code=exited, status=1/FAILURE)

    oct. 30 09:42:58 syslog systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
    oct. 30 09:42:58 syslog systemd[1]: filebeat.service: Failed with result 'exit-code'.
    oct. 30 09:42:58 syslog systemd[1]: filebeat.service: Service hold-off time over, scheduling restart.
    oct. 30 09:42:58 syslog systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
    oct. 30 09:42:58 syslog systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
    oct. 30 09:42:58 syslog systemd[1]: filebeat.service: Start request repeated too quickly.
    oct. 30 09:42:58 syslog systemd[1]: filebeat.service: Failed with result 'exit-code'.
    oct. 30 09:42:58 syslog systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

My environment variable appears well when I do the printenv and env command

When I do this setup:
- "/var/log/SYSLOG-NG/*/*/*/WINDOWS/10.1.251.2/*.log"

There is no error and the logs are being sent so I think it comes from my variable but I don't know why

My .yml file after change

filebeat.inputs:
- type: log
  paths:
    - "/var/log/SYSLOG-NG/${DATE_Y}/${DATE_M}/${DATE_D}/WINDOWS/10.1.251.2/*.log"
  fields:
    log_type: windows
  fields_under_root: true
- TYPE: log
  paths:
    - "/var/log/SYSLOG-NG/${DATE_Y}/${DATE_M}/${DATE_D}/ANTIVIRUS_ESEP/*/*.log"
  fields:
    log_type: esep
  fields_under_root: true

[leandrojmp (Leandro Pereira)]

Setting the variables on a bashrc script won't work if you are running filebeat as a service, the variables set this way will only be available for a login shell, and filebeat does not start a login shell when running.

The best solution in your case, since you are using systemd, is to create an Environment File and configure your service to read the variables from this file.

To do that you can create a file in the directory /etc/systemd/system/filebeat.service.d/ or use systemctl edit filebeat.service to create a file in the same directory, with the following content.

[Service]
EnvironmentFile=/etc/filebeat/envs

This way when filebeat starts it will look for the envs file inside /etc/filebeat/ with environment variables.

In the envs file you will have your variables.

DATE_D=30
DATE_Y=2020
DATE_M=10

You need to run systemctl daemon-reload and systemctl restart filebeat to apply the changes.

But since your variables are dynamic and changes every day, you will need to create a script to change the envs file everyday em restart filebeat when the day changes.

If you don't want to do all that, you can set the variables from your bashrc script, but you will need to run filebeat using the command line.

/path/to/filebeat/bin/filebeat -f /path/to/config/filebeat.yml

Is there any reason for you to not use wildcards in the path?