Env variables in not working in custom fields in filebeat

Filebeat version 5.6
centos 7.0

I am using env variable like this
fields_under_root: true
clustername: ${CLUSTER_NAME}

output of # env

however inspite of setting it as env variable getting following error
CRIT Exiting: error loading states for prospector 0: missing field accessing 'filebeat.prospectors.0.fields.clustername' (source:'/home/cloud-user/rohsing2/testfilebeat/filebeat-5.6.1-linux-x86_64/prospectors/audit-filebeat.yml')

Can you please be more detailed in describing how you have things setup and what commands you are running. Based on what you have described, I was not able to reproduce the issue. See below.

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

# rpm -qa | grep filebeat

# env | grep CLUSTER_NAME

# cat /etc/filebeat/filebeat.yml
- paths: ['/var/log/messages']
    clustername: ${CLUSTER_NAME}

output.console.enabled: true

# filebeat.sh 
{"@timestamp":"2017-09-21T18:10:25.298Z","beat":{"hostname":"beat","name":"beat","version":"5.6.1"},"fields":{"clustername":"lokitest"},"input_type":"log","message":"Sep 21 18:09:19 beat systemd: Removed slice user-0.slice.","offset":54426,"source":"/var/log/messages","type":"log"}

Thanks, I am getting the same error, I am using LINUX 64-BIT 5.6.1 version for filebeat,

I have followed the same steps as above following is the output

 #cat /etc/redhat-release
     CentOS Linux release 7.3.1611 (Core)
#env | grep CLUSTER_NAME
#cat filebeat.yml
- paths: ['/var/log/messages']
    clustername: ${CLUSTER_NAME}

output.console.enabled: true

# sudo ./filebeat -v -c filebeat.yml
Exiting: error loading states for prospector 2639214862234946428: missing field accessing 
'filebeat.prospectors.0.fields.clustername' (source:'filebeat.yml')

It's not the same. You used sudo which does not preserve the environment for security purposes.

When you get to the point of running Filebeat as a service you will be using systemd. When a service starts it receives a clean environment (just like when you used sudo). For systemd you need to configure an override file with the environment data for the service. You can let systemd create the override file for you by running:

systemctl edit filebeat.service

Then configure the overrides for the service.


You can use EnvironmentFile= if you want to point to a file with the environment variables.

1 Like

Thanks Andrew, for nice explanation.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.