Multiple configuration file with enabled option

Elastic Stack Beats
1

I have the following configuration file /etc/filebeat/filebeat.yml:

filebeat.config.prospectors:
    path: /etc/filebeat/conf.d/*.yml

output.logstash:
    hosts: ["host"]

In each yml file in the folder /etc/filebeat/conf.d/ I have the following:

- type: log
  enabled: true
  paths:
    - /var/log/log-file-a.log
  fields:
    ...

- type: log
  enabled: true
  paths:
    - /var/log/log-file-b.log
  fields:
    ...

- type: log
  enabled: true
  paths:
    - /var/log/log-file-c.log
  fields:
    ...

Each file is for different domain. If I want to turn off the monitoring for certain log files of domain or for the whole domain, I set the enabled option to false.

Despite when I set the enabled to false, it still monitors the log files, and sends them to the logstash server. What am I doing wrong?

2

You need to enable reloading (I assume you use Filebeat 5.6 based on your configuration.):

filebeat.config.prospectors:
  path: /etc/filebeat/conf.d/*.yml
  reload.enabled: true
  reload.period: 10s
3

Oh Sorry I forgot to provide the details: my filebeat version outputs:

filebeat version 6.4.2 (amd64), libbeat 6.4.2 [e193f6d68b25b7ddbe3a3ed8d60bc07fea1ef800 built 2018-09-26 12:42:46 +0000 UTC]

4

I followed the instructions, described here: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html.

In the documentation for version 5.6 I see I have the reload.enabled option. But the doc for the current version mentions only scan_frequency option. Should I define it as 0s to achieve the same behavior?

5

There is a subarticle titled "Live reloading": https://www.elastic.co/guide/en/beats/filebeat/current/_live_reloading.html

In 6.4 you can achieve the same thing. Note that prospector is deprecated and I used inputs instead.

filebeat.config.inputs:
  enabled: true
  path: configs/*.yml
  reload.enabled: true
  reload.period: 10s

scan_frequency specifies the interval between checking if the input files have changed. It is unrelated to configuration reload. See more: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-scan-frequency

6

Thank you for your assistance. I adjusted my configuration. Removed the deprecated option and added the reload options.

filebeat.config.inputs:
  enabled: true
  path: /etc/filebeat/conf.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.logstash:
  hosts: ["host"]

I have 4 additional yml files in the folder: /etc/filebeat/conf.d/. Each file belongs to one domain, and in each file, there are 3 log definition (Apache and PHP logs).

Despite that the enabled option is false. It still monitors the files and sends them to the logstash, and eventually I can see them in the ES. What am I doing wrong?

7

Could you please share the debug logs? (./filebeat -e -d "*")

8

Sure,

I pasted it here: https://pastebin.com/660a4gZP

9

I cannot access it, because it is private. You could just paste it here.

10

Ohh sorry, I changed it to the public. I couldn't paste it here, because only 7k lines are available and I wanted to provide a longer log.

11

Should I provide more detailed information about my configuration?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.