Filebeat doesn't work with tail_files for Multiline pattern files

Elastic Stack Beats
1

I am trying with the option of tail_files: true but its not even reading the files which are continuously coming to the directory.

Its opening the prospector but not able to start harvester for any file.

2018/04/04 05:46:57.996815 registrar.go:123: INFO States Loaded from registrar: 0
2018/04/04 05:46:57.996860 crawler.go:38: INFO Loading Prospectors: 2
2018/04/04 05:46:57.996878 registrar.go:236: INFO Starting Registrar
2018/04/04 05:46:57.996979 async.go:72: INFO Start sending events to output
2018/04/04 05:46:57.996973 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/04/04 05:46:57.997045 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/04/04 05:46:57.997423 prospector.go:124: INFO Starting prospector of type: log; id: 16398777863234065337
2018/04/04 05:46:57.997610 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/04/04 05:46:57.997788 prospector.go:124: INFO Starting prospector of type: log; id: 4689995440609842373
2018/04/04 05:46:57.997817 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2018/04/04 05:47:27.994472 metrics.go:39: INFO Non-zero metrics in the last 30s: registrar.writes=1
2018/04/04 05:47:57.994265 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:48:27.994229 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:48:57.994269 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:49:27.994276 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:49:57.994213 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:50:27.994251 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:50:57.994161 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/04/04 05:51:27.994222 metrics.go:34: INFO No non-zero metrics in the last 30s

My configuration is

filebeat.prospectors:

  • input: log
    paths:

    • /backup/OUTPUT_BACKUP/MME_LOG_TRACE/BLN_EXPORT/KOL_HUW_MME/*.log.decode
      tail_files: true
      fields:
      document_type: med_el_east_bln
      node_name: Kolkata
      zone: East
      fields_under_root: true
      multiline.pattern: ^RECORD
      multiline.negate: true
      multiline.match: after
      #close_inactive: 2m
      #ignore_older: 5m
      #clean_inactive: 10m

  • input: log
    paths:

    • /backup/OUTPUT_BACKUP/MME_LOG_TRACE/RDC_EXPORT/KOL_HUW_MME/*.log.decode
      tail_files: true
      fields:
      document_type: med_el_east_rdc
      node_name: Kolkata
      zone: East
      fields_under_root: true
      multiline.pattern: ^RECORD
      multiline.negate: true
      multiline.match: after
      #close_inactive: 2m
      #ignore_older: 5m
      #clean_inactive: 10m
      filebeat.publish_async: true
      #------------------------------- Redis output ----------------------------------
      output.redis:

    Boolean flag to enable or disable the output module.

    #enabled: true

    The list of Redis servers to connect to. If load balancing is enabled, the

    events are distributed to the servers in the list. If one server becomes

    unreachable, the events are distributed to the reachable servers only.

    hosts: ["10.14.162.51:6379"]

    The Redis port to use if hosts does not contain a port number. The default

    is 6379.

    port: 6379

    The name of the Redis list or channel the events are published to. The

    default is filebeat.

    key: "%{[document_type]}"

2

Could you please share the debug logs of Filebeat? You can get them using ./filebeat -e -d "*".

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.