Filebeat prospectors autoreload not working

Elastic Stack Beats
1

Hello All,

Is there any way for auto reloading filebeat config after changing prospector fields? Below is my sample config:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    #- /var/log/*.log
    - C:\ELK\filebeat-6.1.1-windows-x86_64\*.log

  fields:
    level: error
 
filebeat.config:
  prospectors:
    enabled: true
    path: C:\ELK\filebeat-6.1.1-windows-x86_64\*.yml
    reload.enabled: true
    reload.period: 10s
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

I have a requirement of changing field value. If I change prospector field value, filebeat not effecting those changes until restart.

Please help me if we have any alternative to make this work.

-Thanks

2

Hello @Eshwar_Kumar, I think there is a small problem with your configuration if you look at the documentation.

I would remove the following:

filebeat.config:
  prospectors:
    enabled: true
    path: C:\ELK\filebeat-6.1.1-windows-x86_64\*.yml
    reload.enabled: true
    reload.period: 10s

and replace it with

filebeat.config.inputs:
  enabled: true
  path: C:\ELK\filebeat-6.1.1-windows-x86_64\*.yml
  reload.enabled: true
  reload.period: 10s

After you just create a new file in the reloading path, that file must have a yml extension. In the file you just define the prospector.

- type: log
  enabled: true
  paths:
    - C:\ELK\filebeat-6.1.1-windows-x86_64\*.log

  fields:
    level: error

See the load external configuration documentation for more details.

3

Thanks. Correct me if i am wrong. You are saying to create a new yml file which will be relaoded automatically without filebeat restart. But I am looking for a feature where if I change a field value in existing yml file ( changing field "level" value as "warn" in my scenario) that field should get updated for new logs without filebeat restart. I am wondering if we have this feature in filebeat.

  • Thank you
4

@Eshwar_Kumar not everything in the Filebeat configuration support live reload, but prospector related configuration supports live reloading when you define them in externals file.

In the case you are describing updating the field value from error to warn will trigger a reload.

5

I am newbie for this. I seriously not understood filebeat configuration and external configuration. As per my knowledge i am using filebeat.yml and i am running as "filebeat -e -c "filebeat.yml" -d "publish"". In this filebeat.yml i am mentioning prospectors which contains path and fileds to be added. Should i add same prospectors with new field values in external.yml? If not please give me sample filebeat.yml and external.yml files what to be tested. My requirement is just if i change a field value of a prospector, without reload that field value should get updated for my new logs. Please help on this.

  • Thanks
6

@Eshwar_Kumar I gave you the filebeat.yml configuration and the configuration for the external file in the comment above, Like I've said you will remove the prospector definition from the filebeat.yml file and just define them in the external file. When Filebeat is running and you change the external files Filebeat will reload the configuration and use the new prospector definition.

The modification will not change how you start Filebeat you will still use the same command and arguments.

7

Seems I am missing something.

filebeat.yml:

    filebeat.config.inputs:
      enabled: true
      path: C:\ELK\filebeat-6.1.1-windows-x86_64\*.yml
      reload.enabled: true
      reload.period: 10s

1.yml in C:\ELK\filebeat-6.1.1-windows-x86_64\ directory:

- type: log
  enabled: true
  paths:
    - C:\ELK\filebeat-6.1.1-windows-x86_64\1.log

  fields:
    level: error

Started filebeat with filebeat -e -c filebeat.yml -d "publish" . Logs flowing good with field name level and value error.

Now created2.yml in same directory C:\ELK\filebeat-6.1.1-windows-x86_64\ as:
2.yml:

- type: log
  enabled: true
  paths:
    - C:\ELK\filebeat-6.1.1-windows-x86_64\2.log

  fields:
    level: warn

2.log file lines were not flowing in the filebeat. I have changed field value of level in 1.yml from error to debug. Now new logs were still getting value for field level as error not debug.

I followed exactly same as per documents specified and the confs you commented.

I tried filebeat 6.1 and 6.3 as well.

Please suggest me where i am doing mistake.

Thanks.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.