Upgrade 1.3.0 -> 5.6.4: deprecated document_type use fileds

[stefws (Steffen Winther Sørensen)]

Upgraded a filebeat from 1.3.0 to 5.6.4 but it doesn't seem to collect and ship data to logstash. Upon launch filebeat complains i log about deprecated feature document_type in my prospector:

2017-11-10T15:22:37+01:00 WARN DEPRECATED: document_type is deprecated. Use fields instead.
2017-11-10T15:22:37+01:00 INFO Starting prospector of type: log; id: 10372473829951082110 
2017-11-10T15:22:37+01:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-11-10T15:23:07+01:00 INFO Non-zero metrics in the last 30s: registrar.writes=1
2017-11-10T15:23:37+01:00 INFO No non-zero metrics in the last 30s

my prospector:

# cat prospectors.d/dcsstat.yml 
filebeat:
  prospectors:
    -
      paths:
        - /opt/<app server>/log/appserv.stat
      document_type: dcsstat
      fields:
        filter: owmstat

On service start is seems ok:

Starting filebeat: 2017/11/10 14:20:34.402507 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/11/10 14:20:34.402542 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.4
2017/11/10 14:20:34.402587 logstash.go:91: INFO Max Retries set to: 3
2017/11/10 14:20:34.402623 outputs.go:108: INFO Activated logstash as output plugin.
2017/11/10 14:20:34.402681 publish.go:300: INFO Publisher name: dcs3
2017/11/10 14:20:34.402809 async.go:63: INFO Flush Interval set to: 1s
2017/11/10 14:20:34.402827 async.go:64: INFO Max Bulk Size set to: 2048
2017/11/10 14:20:34.402939 metrics.go:23: INFO Metrics logging every 30s
2017/11/10 14:20:34.402952 config.go:114: INFO Additional config files are fetched from: /etc/filebeat/prospectors.d
2017/11/10 14:20:34.403024 config.go:87: INFO Additional configs loaded from: /etc/filebeat/prospectors.d/dcsstat.yml
Config OK
[ OK ]

The registry stay empty:

# cat /var/lib/filebeat/registry 
[]

Is the deprecation warning hinder it to run?

[stefws (Steffen Winther Sørensen)]

Even though I change docment_type to a field so I got now complains at launch, filebeat 5.6.4 doesn't seem to ship any data to my logstash output nor update it's registry.

cat /etc/filebeat/prospectors.d/dcsstat.yml 
filebeat:
  prospectors:
    -
      paths:
        - /opt/<app server>/log/appserv.stat
      #document_type: dcsstat
      fields:
        type: dcsstat
        filter: owmstat

# service filebeat start
Starting filebeat: 2017/11/13 08:24:10.345106 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/11/13 08:24:10.345134 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.4
2017/11/13 08:24:10.345200 logstash.go:91: INFO Max Retries set to: 3
2017/11/13 08:24:10.345221 metrics.go:23: INFO Metrics logging every 30s
2017/11/13 08:24:10.345241 outputs.go:108: INFO Activated logstash as output plugin.
2017/11/13 08:24:10.345293 publish.go:300: INFO Publisher name: dcs3
2017/11/13 08:24:10.345474 async.go:63: INFO Flush Interval set to: 1s
2017/11/13 08:24:10.345487 async.go:64: INFO Max Bulk Size set to: 2048
2017/11/13 08:24:10.345632 config.go:114: INFO Additional config files are fetched from: /etc/filebeat/prospectors.d
2017/11/13 08:24:10.345680 config.go:87: INFO Additional configs loaded from: /etc/filebeat/prospectors.d/dcsstat.yml
Config OK
                                                           [  OK  ]
# tail /var/log/beats/filebeat.log
2017-11-13T09:24:10+01:00 INFO Loading registrar data from /var/lib/filebeat/registry
2017-11-13T09:24:10+01:00 INFO States Loaded from registrar: 0
2017-11-13T09:24:10+01:00 INFO Loading Prospectors: 1
2017-11-13T09:24:10+01:00 INFO Starting Registrar
2017-11-13T09:24:10+01:00 INFO Start sending events to output
2017-11-13T09:24:10+01:00 INFO Prospector with previous states loaded: 0
2017-11-13T09:24:10+01:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-11-13T09:24:10+01:00 INFO Starting prospector of type: log; id: 11876712458280652192 
2017-11-13T09:24:10+01:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-11-13T09:24:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:25:10+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:25:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:26:10+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:26:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:27:10+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:27:40+01:00 INFO No non-zero metrics in the last 30s
2017-11-13T09:28:10+01:00 INFO No non-zero metrics in the last 30s

File path /opt//log/appserv.stat is a symlink to latest log per day but it worked just fine with v.1.3

Changing it directly to a log file instead of through a symlink makes it work. Is this a bug or feature degration?
Using file patterns and ignore_older,close_,clean_ or enable symlinks option should fix this

[steffens (Steffen Siering)]

Supporting symlinks out of the box was an oversight in filebeat versions 1.x. Using symlinks is not recommended, but if you really want to use symlinks, have a look at the symlinks setting(https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_symlinks).

Is the deprecation warning hinder it to run?

The deprecation warning is no big problem, as filebeat will continue to run. But with 6.0 support for _type is removed/deprecated in Elasticsearch. If you still want a type field, you can use fields.