Beats ignores configuration yml

Elastic Stack Beats
1

My yml file validates and specifically uses a non-default logging directory along with other settings. The logs are put in the default locations along with other unexpected default behavior given my yml.

Any ideas?

filebeat.prospectors:
- paths:
    - /var/log/some_service/json/events*.json
  document_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.logstash:
  hosts: ['192.168.0.47:6000']
  
  logging.level: info
  logging.to_files: true
  logging.to_syslog: false
  logging.files:
    path: /var/log/filebeat/
    name: some_service_filebeat.log
    keepfiles: 7

some-pc:filebeat-5.5.2-darwin-x86_64 johndoe$ ./filebeat -configtest -e

2017/09/07 00:23:41.838968 beat.go:285: INFO Home path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64] Config path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64] Data path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64/data] Logs path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64/logs]
2017/09/07 00:23:41.839012 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.2
2017/09/07 00:23:41.839101 logstash.go:90: INFO Max Retries set to: 3
2017/09/07 00:23:41.839112 metrics.go:23: INFO Metrics logging every 30s
2017/09/07 00:23:41.839216 outputs.go:108: INFO Activated logstash as output plugin.
2017/09/07 00:23:41.839318 publish.go:295: INFO Publisher name: some-pc.local
2017/09/07 00:23:41.839445 async.go:63: INFO Flush Interval set to: 1s
2017/09/07 00:23:41.839456 async.go:64: INFO Max Bulk Size set to: 2048
2

I don't know if your tab-spacing is an issue in your case or it is pasted like that in the post - yml is a bit serious about it.

try removing spaces at the start of the line for logging.level, to_files, etc.

I didn't run into this issue - i am guessing from my experience with spacing issues in yml.

3

Hi Mmsasidhar,

The configtest parameter validates the yaml spacing as part of its validation. You can see the output of that command above. I'm sure that is not the problem in this case.

4

Configtest - sounds right! Hope you will figure out the solution soon. All the best.

I will test on Monday and get back if it's open even then.

5

It's indeed the spacing. configtest is not "perfect" yet. The sample configuration file clearly shows the logging settings being put into the output.logstash section.

Try this config:

filebeat.prospectors:
- paths:
    - /var/log/some_service/json/events*.json
  document_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.logstash:
  hosts: ['192.168.0.47:6000']
  
logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/filebeat/
  name: some_service_filebeat.log
  keepfiles: 7
6

Some updates...

The configtest option will raise an error over a tab which gives the false impression it actually is validating whitespace (or even valid yaml). It would be better not to have this than to have it telling customers the configs are valid when they aren't.

Thank you for the observation and suggestion. I did move the logging config all the way to the left margin as you suggest. I still don't get any output in the configured folder. Also Filebeat still lists the default directories at startup regardless of my config:

2017/09/08 17:06:18.607786 beat.go:285: INFO Home path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64] Config path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64] Data path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64/data] Logs path: [/Users/johndoe/devel/apps/filebeat-5.5.2-darwin-x86_64/logs]
2017/09/08 17:06:18.607816 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.2
2017/09/08 17:06:18.607872 logstash.go:90: INFO Max Retries set to: 3
2017/09/08 17:06:18.607915 metrics.go:23: INFO Metrics logging every 30s
2017/09/08 17:06:18.607929 outputs.go:108: INFO Activated logstash as output plugin.
7

The paths printed are due to the default directory layout of beats. Check out the Paths documentation. The logging.files.path sets an alternative path for the logger instance. The default log output is set to be logging.files.path: ${paths.logs}

8

This topic was automatically closed after 21 days. New replies are no longer allowed.