Hi all,
I installed this new environment:
SERVER (loststash+elasticsearch+kibana)
- logstash 6.5.1
- elasticsearch 6.5.1
- kibana 6.5.1
REMOTE MACHINE (generating logs)
- filebeat 5.0.1
I passed to filebeat our old logs, restarted the service and this is the result:
2018-11-30T09:53:53+01:00 DBG Try to publish 835 events to logstash with window size 608
2018-11-30T09:53:54+01:00 DBG update max ok window size: 405 < 608
2018-11-30T09:53:54+01:00 DBG increase window size to: 912
2018-11-30T09:53:54+01:00 DBG set to batchSize: 835
2018-11-30T09:53:54+01:00 DBG 608 events out of 835 events sent to logstash. Continue sending
2018-11-30T09:53:54+01:00 DBG Try to publish 227 events to logstash with window size 835
2018-11-30T09:53:54+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:53:54+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:53:54+01:00 DBG 227 events out of 227 events sent to logstash. Continue sending
2018-11-30T09:53:54+01:00 DBG send completed
2018-11-30T09:53:58+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:53:58+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:06+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:06+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:16+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:16+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:21+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=84278 filebeat.harvester.started=82 filebeat.harvester.running=82 libbeat.logstash.published_and_acked_events=2026 libbeat.logstash.publish.read_bytes=72 libbeat.publisher.published_events=2026 filebeat.harvester.open_files=82 libbeat.logstash.call_count.PublishEvents=1
2018-11-30T09:54:26+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:26+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:36+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:36+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:46+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:46+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:54:51+01:00 INFO No non-zero metrics in the last 30s
2018-11-30T09:54:56+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:54:56+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:55:06+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:55:06+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:55:16+01:00 DBG End of file reached: /var/log/app.20181129221005.log; Backoff now.
2018-11-30T09:55:16+01:00 DBG End of file reached: /var/log/app.20180418221219.log; Backoff now.
2018-11-30T09:55:21+01:00 INFO No non-zero metrics in the last 30s
Kibana is indexing but only for few minus:
How can i let it process all my logs?
This is my filebeat config file:
filebeat:
prospectors:
-
paths:
- /var/log/app.*.log
ignore_older: 0
output:
logstash:
hosts: ["XXX:XXX:XXX:XXX:5044"]
console:
pretty: true
logging:
to_syslog: true
to_files: true
files:
path: /var/log/foo/filebeat-log
name: filebeat.log
rotateeverybytes: 10485760
keepfiles: 7
level: debug