Filebeat - input not enabled when using external config load

Elastic Stack Beats
1

Hi There,

I'm trying to use filebeat to import logs from custom locations. In configuration I have:

   filebeat:
     config.inputs:
       enabled: true
       path: "/etc/filebeat/conf.d/*.yml"
   output.logstash:
    hosts:
      - "localhost:5044"

and in conf.d file I have:

    - type: log
      close_eof: true
      paths:
       - ${LOG_PATH}

Real config is more complex but this should be enough to replicate issue.
When running filebeat: filebeat -e run --once in output I can see:

        DEBUG   [cfgfile]       cfgfile/reload.go:132   Checking module configs from: /etc/filebeat/conf.d/*.yml
        DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /etc/filebeat/conf.d/config.yml

so, config seems to be fine, external input was loaded, but then I can see:

           INFO    [crawler]       beater/crawler.go:71    **Loading Inputs: 0**
           INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. **Enabled inputs: 0**

I was trying various combinations but with no results, all time there is 0 inputs enabled.
However, when use the same settings but defined in one file, without external config. It works as expected.
Can anyone look at it ? Is it a bug or configuration mistake ?

Thanks,

2

Hi eMDee,

Sorry for the delay in getting back to you. If you are still experiencing this issue, could you share the filebeat and logstash version you are using? Which filebeat module are you currently using? If you are not sure you can run sudo filebeat modules list on the host and list the Enabled modules. Could you provide the module's configuration that are enabled?

Thanks,
George

3

Hi George

Thanks for reply, filebeat version is:

filebeat version 6.8.8 (amd64), libbeat 6.8.8 [ecd273d59ab89c70355504b89445563e9a987812 built 2020-03-18 22:26:53 +0000 UTC]

Logstash is also 6.8.8
Im not using any modules:

filebeat modules list

Enabled:

Disabled:

4

Hi eMDee,

Where/how are you specifying the environment variable ${LOG_PATH}? I think this could be a syntax issue or a hidden space within the yml/conf.d file. Depending on your text editor you can see if there are any spaces that could cause a parse error?

5

Hi George,

Variable LOG_PATH is set right before filebeat run and seems to be fine. When filebeat input is set in the same file (filebeat.yml) it works as expected. Anyway, I replaced variable by specific path and I can observe the same behaviour.

	INFO	[crawler]	beater/crawler.go:71	Loading **Inputs: 0**
	INFO	log/input.go:157	Configured paths: [/tmp/log.log]
	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. **Enabled inputs: 0**
	INFO	log/input.go:157	Configured paths: [/tmp/log.log]
	INFO	cfgfile/reload.go:224	Loading of config files completed.
	INFO	beater/filebeat.go:447	Running filebeat once. Waiting for completion ...
	INFO	beater/filebeat.go:449	All data collection completed. Shutting down.
        INFO	beater/crawler.go:148	Stopping Crawler
	INFO	beater/crawler.go:158	Stopping **0 inputs**

As additional information,
I check also other filebeat versions up to 7.9.1 - same results, no inputs enable when using external configuration files.

6

Hi eMDee,

Looking from the info you provided I think its a syntax/space issue as there are no inputs being listed. Could you possibly turn on logging.level: debug in your filebeat.yml config and display the logs?

Thanks,
George

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.