Env variables in not working in custom fields in filebeat

Elastic Stack Beats
1

Filebeat version 5.6
centos 7.0

I am using env variable like this
fields_under_root: true
fields:
clustername: ${CLUSTER_NAME}

output of # env
HOSTNAME=gsljumphost.cisco.com
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
CLUSTER_NAME=lokitest

however inspite of setting it as env variable getting following error
CRIT Exiting: error loading states for prospector 0: missing field accessing 'filebeat.prospectors.0.fields.clustername' (source:'/home/cloud-user/rohsing2/testfilebeat/filebeat-5.6.1-linux-x86_64/prospectors/audit-filebeat.yml')

2

Can you please be more detailed in describing how you have things setup and what commands you are running. Based on what you have described, I was not able to reproduce the issue. See below.

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

# rpm -qa | grep filebeat
filebeat-5.6.1-1.x86_64

# env | grep CLUSTER_NAME
CLUSTER_NAME=lokitest

# cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- paths: ['/var/log/messages']
  fields:
    clustername: ${CLUSTER_NAME}

output.console.enabled: true

# filebeat.sh 
{"@timestamp":"2017-09-21T18:10:25.298Z","beat":{"hostname":"beat","name":"beat","version":"5.6.1"},"fields":{"clustername":"lokitest"},"input_type":"log","message":"Sep 21 18:09:19 beat systemd: Removed slice user-0.slice.","offset":54426,"source":"/var/log/messages","type":"log"}
3

Thanks, I am getting the same error, I am using LINUX 64-BIT 5.6.1 version for filebeat,

I have followed the same steps as above following is the output

 #cat /etc/redhat-release
     CentOS Linux release 7.3.1611 (Core)
#env | grep CLUSTER_NAME
     CLUSTER_NAME=lokitest
#cat filebeat.yml
filebeat.prospectors:
- paths: ['/var/log/messages']
  fields:
    clustername: ${CLUSTER_NAME}

output.console.enabled: true

#command
# sudo ./filebeat -v -c filebeat.yml
Exiting: error loading states for prospector 2639214862234946428: missing field accessing 
'filebeat.prospectors.0.fields.clustername' (source:'filebeat.yml')
4

It's not the same. You used sudo which does not preserve the environment for security purposes.

When you get to the point of running Filebeat as a service you will be using systemd. When a service starts it receives a clean environment (just like when you used sudo). For systemd you need to configure an override file with the environment data for the service. You can let systemd create the override file for you by running:

systemctl edit filebeat.service

Then configure the overrides for the service.

[Service]
Environment=CLUSTER_NAME=lokitest

You can use EnvironmentFile= if you want to point to a file with the environment variables.

1 Like
5

Thanks Andrew, for nice explanation.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.