Upgraded a filebeat from 1.3.0 to 5.6.4 but it doesn't seem to collect and ship data to logstash. Upon launch filebeat complains i log about deprecated feature document_type in my prospector:
2017-11-10T15:22:37+01:00 WARN DEPRECATED: document_type is deprecated. Use fields instead.
2017-11-10T15:22:37+01:00 INFO Starting prospector of type: log; id: 10372473829951082110
2017-11-10T15:22:37+01:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-11-10T15:23:07+01:00 INFO Non-zero metrics in the last 30s: registrar.writes=1
2017-11-10T15:23:37+01:00 INFO No non-zero metrics in the last 30s
Starting filebeat: 2017/11/10 14:20:34.402507 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/11/10 14:20:34.402542 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.4
2017/11/10 14:20:34.402587 logstash.go:91: INFO Max Retries set to: 3
2017/11/10 14:20:34.402623 outputs.go:108: INFO Activated logstash as output plugin.
2017/11/10 14:20:34.402681 publish.go:300: INFO Publisher name: dcs3
2017/11/10 14:20:34.402809 async.go:63: INFO Flush Interval set to: 1s
2017/11/10 14:20:34.402827 async.go:64: INFO Max Bulk Size set to: 2048
2017/11/10 14:20:34.402939 metrics.go:23: INFO Metrics logging every 30s
2017/11/10 14:20:34.402952 config.go:114: INFO Additional config files are fetched from: /etc/filebeat/prospectors.d
2017/11/10 14:20:34.403024 config.go:87: INFO Additional configs loaded from: /etc/filebeat/prospectors.d/dcsstat.yml
Config OK
[ OK ]
Even though I change docment_type to a field so I got now complains at launch, filebeat 5.6.4 doesn't seem to ship any data to my logstash output nor update it's registry.
cat /etc/filebeat/prospectors.d/dcsstat.yml
filebeat:
prospectors:
-
paths:
- /opt/<app server>/log/appserv.stat
#document_type: dcsstat
fields:
type: dcsstat
filter: owmstat
# service filebeat start
Starting filebeat: 2017/11/13 08:24:10.345106 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/11/13 08:24:10.345134 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.4
2017/11/13 08:24:10.345200 logstash.go:91: INFO Max Retries set to: 3
2017/11/13 08:24:10.345221 metrics.go:23: INFO Metrics logging every 30s
2017/11/13 08:24:10.345241 outputs.go:108: INFO Activated logstash as output plugin.
2017/11/13 08:24:10.345293 publish.go:300: INFO Publisher name: dcs3
2017/11/13 08:24:10.345474 async.go:63: INFO Flush Interval set to: 1s
2017/11/13 08:24:10.345487 async.go:64: INFO Max Bulk Size set to: 2048
2017/11/13 08:24:10.345632 config.go:114: INFO Additional config files are fetched from: /etc/filebeat/prospectors.d
2017/11/13 08:24:10.345680 config.go:87: INFO Additional configs loaded from: /etc/filebeat/prospectors.d/dcsstat.yml
Config OK
[ OK ]
# tail /var/log/beats/filebeat.log
2017-11-13T09:24:10+01:00 INFO Loading registrar data from /var/lib/filebeat/registry
2017-11-13T09:24:10+01:00 INFO States Loaded from registrar: 0
2017-11-13T09:24:10+01:00 INFO Loading Prospectors: 1
2017-11-13T09:24:10+01:00 INFO Starting Registrar
2017-11-13T09:24:10+01:00 INFO Start sending events to output
2017-11-13T09:24:10+01:00 INFO Prospector with previous states loaded: 0
2017-11-13T09:24:10+01:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-11-13T09:24:10+01:00 INFO Starting prospector of type: log; id: 11876712458280652192
2017-11-13T09:24:10+01:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-11-13T09:24:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:25:10+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:25:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:26:10+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:26:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:27:10+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:27:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:28:10+01:00 INFO No non-zero metrics in the last 30s
File path /opt//log/appserv.stat is a symlink to latest log per day but it worked just fine with v.1.3
Changing it directly to a log file instead of through a symlink makes it work. Is this a bug or feature degration?
Using file patterns and ignore_older,close_,clean_ or enable symlinks option should fix this
Supporting symlinks out of the box was an oversight in filebeat versions 1.x. Using symlinks is not recommended, but if you really want to use symlinks, have a look at the symlinks setting(Configure inputs | Filebeat Reference [8.11] | Elastic).
Is the deprecation warning hinder it to run?
The deprecation warning is no big problem, as filebeat will continue to run. But with 6.0 support for _type is removed/deprecated in Elasticsearch. If you still want a type field, you can use fields.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.