Environment variables not replaced in external configuration file

Elastic Stack Beats
1

I face the problem that environment variables in external config files are not replaced. The external config files are imported to the filebeat.yml file via the filebeat.config.prospectors option.

My external config file contains some environment variables. These are not replaced. If I copy the contents of the external yml-file into the file filebeat.yml (option filebeat.prospectors) everything works fine. Does anyone have any idea why this may be or am I doing something wrong?

Not working:
filebeat.yml

    path.home: ${FILEBEAT_HOME:?You need to set the FILEBEAT_HOME environment variable}
    path.config: ${INSTANCE_HOME:?You need to set the INSTANCE_HOME environment variable}
    path.logs: ${LOGS_PATH:?You need to set the LOGS_PATH environment variable}
    path.data: ${INSTANCE_HOME:?You need to set the INSTANCE_HOME environment variable}/data
    custom.env: ${CUSTOM_ENV:?You need to set the CUSTOM_ENV environment variable}
    custom.env.enabled: ${CUSTOM_ENV_ENABLED:?You need to set the CUSTOM_ENV_ENABLED environment variable}

    filebeat.config.prospectors:
      enabled: true
      path: configs/*.yml
    .... further configuration options ommited

config/example1.yml

    - type: log
      paths:
        - '/all_logs/${custom.env}_Nodes/*/appl_logs/*.log'
      exclude_files: ['\.gz$']
      encoding: us-ascii
      multiline.pattern: '^[0-9]{2}[/\.:][0-9]{2}[/\.:][0-9]{2,4}'
      multiline.negate: true
      multiline.match: after
      enabled: ${custom.env.enabled}
      fields:
        custom.env: ${custom.env}
      fields_under_root: true
      ignore_older: 48h
      close_inactive: 24h
      clean_inactive: 72h

Working:
filebeat.yml

    path.home: ${FILEBEAT_HOME:?You need to set the FILEBEAT_HOME environment variable}
    path.config: ${INSTANCE_HOME:?You need to set the INSTANCE_HOME environment variable}
    path.logs: ${LOGS_PATH:?You need to set the LOGS_PATH environment variable}
    path.data: ${INSTANCE_HOME:?You need to set the INSTANCE_HOME environment variable}/data
    custom.env: ${CUSTOM_ENV:?You need to set the CUSTOM_ENV environment variable}
    custom.env.enabled: ${CUSTOM_ENV_ENABLED:?You need to set the CUSTOM_ENV_ENABLED environment variable}

    filebeat.prospectors:
    - type: log
      paths:
        - "/all_logs/${custom.env}_Nodes/*/appl_logs/*.log"
      exclude_files: ['\.gz$']
      encoding: us-ascii
      multiline.pattern: '^[0-9]{2}[/\.:][0-9]{2}[/\.:][0-9]{2,4}'
      multiline.negate: true
      multiline.match: after
      enabled: ${custom.env.enabled:false}
      fields:
        custom.env: ${custom.env}
      fields_under_root: true
      ignore_older: 48h
      close_inactive: 24h
      clean_inactive: 72h

I'm using Version 6.2.2 on Linux. Environment variables are passed to filebeat via -E command line option: "./filebeat -E CUSTOM_ENV=exampe1 -E CUSTOM_ENV_ENABLED=true ..."

2

Could you try what happens if you set custom.env inside example1.yml?

BTW I reformatted your post a bit to make it more readable.

3

Unfortunatelly this doesn't work either.

Here's my modified example1.yml:

custom.env: ${CUSTOM_ENV:?You need to set the CUSTOM_ENV environment variable}
custom.env.enabled: ${CUSTOM_ENV_ENABLED:?You need to set the CUSTOM_ENV_ENABLED environment variable}

- type: log
  paths:
    - '/all_logs/${custom.env}_Nodes/*/appl_logs/*.log'
  exclude_files: ['\.gz$']
  encoding: us-ascii
  multiline.pattern: '^[0-9]{2}[/\.:][0-9]{2}[/\.:][0-9]{2,4}'
  multiline.negate: true
  multiline.match: after
  enabled: ${custom.env.enabled}
  fields:
    custom.env: ${custom.env}
  fields_under_root: true
  ignore_older: 48h
  close_inactive: 24h
  clean_inactive: 72h

In the log i see the following message:

2018-03-21T08:31:19.548+0100    ERROR   instance/beat.go:667    Exiting: 1 error: invalid config: yaml: line 3: did not find expected key

I think it references the following line:

  - '/all_logs/${custom.env}_Nodes/*/appl_logs/*.log'
4

Hello @oetzi, This is a bug it and this scenario should be supported in external config, I was able to reproduce it with a simple test case and I have created https://github.com/elastic/beats/issues/6660 to follow up.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.