Clarifications regarding reload configuration

Elastic Stack Beats
1

Based on the docs

You can configure Filebeat to dynamically reload prospector configuration files when there are changes. To do this, you specify a path (Glob) to watch for prospector configuration changes. When the files found by the Glob change, new prospectors are started/stopped according to changes in the configuration files.

What does the changes in the docs refer to? Does it mean new yml files being added to the directory or changes within the existing files itself.

My current config looks something like that

filebeat.config_dir: /usr/share/filebeat/config

filebeat.config.prospectors:
  path: /usr/share/filebeat/config/*.yml
  reload.enabled: true
  reload.period: 60s

and I store a single app_prospectors.yml in /usr/share/filebeat/config.

However, we run a service that generates a new app_prospectors.yml based on the Docker containers that are currently running on the host every 60s. I noticed two things that seemed a little odd:

  1. Filebeat doesn't pick up the new app_prospectors.yml file if it was started when the file isn't present.
  2. Also it does seem to me that changes in my app_prospectors.yml file isn't picked up at all. Not sure how to debug this here :thinking:

I'm running Filebeat v5.6.3.

Sample of my app_prospectors.yml file

---
filebeat.prospectors:
- input_type: log
  paths:
  - "/var/docker/shared/dev/var-log/nginx/access.log*"
  exclude_files:
  - ".gz$"
  document_type: nginx-access
- input_type: log
  paths:
  - "/var/docker/shared/dev/var-log/nginx/error.log*"
  exclude_files:
  - ".gz$"
  document_type: nginx-error
- input_type: log
  paths:
  - "/var/docker/shared/dev/log/rails/production.log*"
  exclude_files:
  - ".gz$"
  document_type: rails
2

Hi @tgxworld,

It should pick both new files and changes. Please review file permissions for your config folder, they should be the same as in filebeat.yml. Also check the log for details in case that doesn't help, paste it here if permissions don't help :slight_smile:

3

Permission looks OK

-rwx------    0 root     root           469 Nov  2 09:29 filebeat.yml
drwx------    2 root     root          4096 Nov  3 12:43 config
-rw-r--r--    1 root     root           437 Nov  3 12:43 app_prospectors.yml

Here are the logs even after I deleted and added a new input_type in app_prospectors.yml

2017/11/03 12:44:28.805873 reload.go:205: INFO Stopping 0 runners ...
2017/11/03 12:44:28.805884 reload.go:193: INFO Starting 0 runners ...
2017/11/03 12:44:58.468330 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.config.reloads=1 libbeat.logstash.call_count.PublishEvents=6 libbeat.logstash.publish.read_bytes=36 libbeat.logstash.publish.write_bytes=5838 libbeat.logstash.published_and_acked_events=50 libbeat.publisher.published_events=50 publish.events=50 registrar.states.update=50 registrar.writes=6
2017/11/03 12:45:28.468405 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=6 libbeat.logstash.publish.read_bytes=36 libbeat.logstash.publish.write_bytes=4908 libbeat.logstash.published_and_acked_events=44 libbeat.publisher.published_events=44 publish.events=44 registrar.states.update=44 registrar.writes=6
2017/11/03 12:45:28.806190 reload.go:205: INFO Stopping 0 runners ...
2017/11/03 12:45:28.806229 reload.go:193: INFO Starting 0 runners ...
2017/11/03 12:45:58.468305 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.config.reloads=1 libbeat.logstash.call_count.PublishEvents=6 libbeat.logstash.publish.read_bytes=36 libbeat.logstash.publish.write_bytes=7531 libbeat.logstash.published_and_acked_events=60 libbeat.publisher.published_events=60 publish.events=60 registrar.states.update=60 registrar.writes=6

If it picks up a new config, will it log the following output like how it did when I start Filebeat?

2017/11/03 02:47:28.493602 log.go:91: INFO Harvester started for file: /var/docker/shared/dev/var-log/nginx/error.log.1
4

Could you also check permissions inside config folder?

5 
/usr/share/filebeat # ls -lah
total 15916
drwxr-xr-x    7 root     root        4.0K Nov  6 07:09 .
drwxr-xr-x   10 root     root        4.0K Nov  6 07:09 ..
-rw-r--r--    1 root     root          41 Oct  6 20:27 .build_hash.txt
-rw-r--r--    1 root     root       94.8K Oct  6 20:27 NOTICE
-rw-r--r--    1 root     root         811 Oct  6 20:27 README.md
drwxr-xr-x    2 root     root        4.0K Nov  6 08:17 config
drwxr-x---    2 root     root        4.0K Nov  7 02:14 data
-rwxr-xr-x    1 root     root       15.3M Oct  6 20:28 filebeat
-rw-r--r--    1 root     root       36.9K Oct  6 20:28 filebeat.full.yml
-rw-r--r--    1 root     root       24.5K Oct  6 20:28 filebeat.template-es2x.json
-rw-r--r--    1 root     root       19.6K Oct  6 20:28 filebeat.template-es6x.json
-rw-r--r--    1 root     root       19.6K Oct  6 20:28 filebeat.template.json
-rwx------    1 root     root         432 Nov  3 04:01 filebeat.yml
drwxr-xr-x    7 root     root        4.0K Nov  3 02:40 module
drwxr-xr-x    2 root     root        4.0K Nov  3 02:40 scripts
/usr/share/filebeat # ls -lah config/
total 16
drwxr-xr-x    2 root     root        4.0K Nov  6 08:17 .
drwxr-xr-x    7 root     root        4.0K Nov  6 07:09 ..
-rw-r--r--    1 root     root         576 Nov  6 08:17 app_prospectors.yml
-rw-r--r--    1 root     root          29 Nov  6 08:16 app_prospectors.yml.old

Here are the permissions.

6

Permissions are wrong for:

Try running chmod go-r config/app_prospectors.yml

7

Sorry for the late reply. Does this mean that the file should belong to the filebeat user and only be readable/writable by the user only?

Because chmod go-r would result in -rw-------

8

Yes, as a security measure we enforce that by default, you can change that behavior by using --strict.perms=false argument, although that's discouraged.

Best regards

9

Awesome thank you for the help!