Filebeat.yml not picking up env variables

Elastic Stack Beats
1
  • I have a filebeat.yml file as written below and it's not taking up the env variable. I get error like this:

Exiting: error unpacking config data: missing field accessing 'name' (source:'/etc/filebeat/filebeat.yml') ...fail!

  • The filebeat.yml has following contents:
name: ${SERVICE_ID}
setup.template.name: ${SERVICE_ID}
setup.template.pattern: ${SERVICE_ID}
output.elasticsearch.index: ${SERVICE_ID}
filebeat.prospectors:
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/info*.log
    fields:
      level: info
    json:
      keys_under_root: true
      overwrite_keys: true
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/system*.log
    fields:
      level: system
    json:
      keys_under_root: true
      overwrite_keys: true
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/error*.log
    fields:
      level: error
    json:
      keys_under_root: true
      overwrite_keys: true
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/api_success*.log
    fields:
      level: api_success
    json:
      keys_under_root: true
      overwrite_keys: true
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/api_error*.log
    fields:
      level: api_error
    json:
      keys_under_root: true
      overwrite_keys: true
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/app_info*.log
    fields:
      level: app_info
    json:
      keys_under_root: true
      overwrite_keys: true
  - type: log
    enabled: true
    paths:
      - ${LOGS_PATH}/app_error*.log
    fields:
      level: app_error
    json:
      keys_under_root: true
      overwrite_keys: true
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# more contents 
...
  • The bashfile which runs filebeat is as following:

#!/bin/bash

SERVICE_ID="xxxx-service"
container_id=`hostname`

export ENV=$1
export NODE_ENV=$1
export SERVER_ENV=$1
export PACKMAN_DEPLOYMENT_TAG=$2
export PACKMAN_INSTANCE_TAG="$container_id"
export ELASTIC_APM_SERVICE_NAME="$SERVICE_ID-$ENV"
export ELASTIC_APM_ENABLED=true

# Creating the log directory
LOGS_PATH="$( pwd )/logs/$SERVICE_ID"

nowTime=$( date +"%Y-%m-%d_%H-%M-%S" )
runLogDir="$LOGS_PATH"/logs-run/$container_id"_"$nowTime
mkdir -p "$runLogDir"
          
# FluentD Requirements
export LOG_PATH="$runLogDir"
export SERVICE_ID="$SERVICE_ID"

echo "Before Copy, $SERVICE_ID, $LOG_PATH"
cp /usr/src/app/node_modules/openapi-rpc-node/filebeat.yml /etc/filebeat/
echo "After Copy, $SERVICE_ID, $LOG_PATH"
service filebeat start
echo "After filebeat Start, $SERVICE_ID, $LOG_PATH"
# td-agent service restart
sh td-agent/td_agent.start.sh $LOG_PATH $ENV
    
node ./server.js

  • This file replaces the filebeat.yml from a desired location having contents stated above to `/etc/filebeat/filebeat.yml' and only then starts filebeat as a service. This script sets up env variables and echos it, which we can see in console alright.

  • We're missing something obvious here. The only solution remaining is to 'awk' and replace placeholders in yml file. But we wanted to use standard and defined solution of env replacement first.

2

As you start filebeat as a service my assumption is that the environment in which filebeat is started does not have access to the environment variables you set inside the script.

Can you try to run filebeat directly in the script instead of as a service and see if it works?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.