Fresh installation apparently imports logs for few seconds and then stops

Elastic Stack Beats
1

Hi all,

I installed this new environment:

SERVER (loststash+elasticsearch+kibana)

  • logstash 6.5.1
  • elasticsearch 6.5.1
  • kibana 6.5.1

REMOTE MACHINE (generating logs)

  • filebeat 5.0.1

I passed to filebeat our old logs, restarted the service and this is the result:
2018-11-30T09:53:53+01:00 DBG Try to publish 835 events to logstash with window size 608
2018-11-30T09:53:54+01:00 DBG update max ok window size: 405 < 608
2018-11-30T09:53:54+01:00 DBG increase window size to: 912
2018-11-30T09:53:54+01:00 DBG set to batchSize: 835
2018-11-30T09:53:54+01:00 DBG 608 events out of 835 events sent to logstash. Continue sending
2018-11-30T09:53:54+01:00 DBG Try to publish 227 events to logstash with window size 835
2018-11-30T09:53:54+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:53:54+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:53:54+01:00 DBG 227 events out of 227 events sent to logstash. Continue sending
2018-11-30T09:53:54+01:00 DBG send completed
2018-11-30T09:53:58+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:53:58+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:06+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:06+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:16+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:16+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:21+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=84278 filebeat.harvester.started=82 filebeat.harvester.running=82 libbeat.logstash.published_and_acked_events=2026 libbeat.logstash.publish.read_bytes=72 libbeat.publisher.published_events=2026 filebeat.harvester.open_files=82 libbeat.logstash.call_count.PublishEvents=1
2018-11-30T09:54:26+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:26+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:36+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:36+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:46+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:46+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:51+01:00 INFO No non-zero metrics in the last 30s
2018-11-30T09:54:56+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:56+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:55:06+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:55:06+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:55:16+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:55:16+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:55:21+01:00 INFO No non-zero metrics in the last 30s

Kibana is indexing but only for few minus:

[Album] Imgur: The magic of the Internet

How can i let it process all my logs?

This is my filebeat config file:

filebeat:

  prospectors:

    -
      paths:
        - /var/log/app.*.log

      ignore_older: 0

output:
  logstash:
    hosts: ["XXX:XXX:XXX:XXX:5044"]
  console: 
    pretty: true

logging:

  to_syslog: true
  to_files: true

  files:

    path: /var/log/foo/filebeat-log
    name: filebeat.log
    rotateeverybytes: 10485760    
    keepfiles: 7

  level: debug
2

According to logs filebeat did reach the end of file. Is this all logs?

Why such an old filebeat? I think 5.0 is EOL for quite some time.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.