Hi all, I am trying to send log files from filebeat (5.5.1) to logstash-docker-elk (repo) (5.3.0) but here's what I am getting in filebeat:
2017-08-23T13:20:57+02:00 INFO Metrics logging every 30s
2017-08-23T13:20:57+02:00 INFO Publisher name: ibrahimsharaf-Inspiron-5559
2017-08-23T13:20:57+02:00 INFO Flush Interval set to: 1s
2017-08-23T13:20:57+02:00 INFO Max Bulk Size set to: 2048
2017-08-23T13:20:57+02:00 INFO filebeat start running.
2017-08-23T13:20:57+02:00 INFO Registry file set to: /var/lib/filebeat/registry
2017-08-23T13:20:57+02:00 INFO Loading registrar data from /var/lib/filebeat/registry
2017-08-23T13:20:57+02:00 INFO States Loaded from registrar: 16
2017-08-23T13:20:57+02:00 INFO Loading Prospectors: 1
2017-08-23T13:20:57+02:00 INFO Starting Registrar
2017-08-23T13:20:57+02:00 INFO Start sending events to output
2017-08-23T13:20:57+02:00 INFO Prospector with previous states loaded: 0
2017-08-23T13:20:57+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-08-23T13:20:57+02:00 INFO Starting prospector of type: log; id: 14525101027454023492
2017-08-23T13:20:57+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-08-23T13:20:57+02:00 INFO Harvester started for file: /var/log/filebeat/logs/03012017.log
2017-08-23T13:20:57+02:00 INFO Harvester started for file: /var/log/filebeat/logs/09052017.log
2017-08-23T13:20:57+02:00 ERR Failed to publish events caused by: EOF
2017-08-23T13:20:57+02:00 INFO Error publishing events (retrying): EOF
2017-08-23T13:20:58+02:00 ERR Failed to publish events caused by: EOF
2017-08-23T13:20:58+02:00 INFO Error publishing events (retrying): EOF
2017-08-23T13:21:00+02:00 ERR Failed to publish events caused by: EOF
2017-08-23T13:21:00+02:00 INFO Error publishing events (retrying): EOF
2017-08-23T13:21:04+02:00 ERR Failed to publish events caused by: EOF
2017-08-23T13:21:04+02:00 INFO Error publishing events (retrying): EOF
2017-08-23T13:21:12+02:00 ERR Failed to publish events caused by: EOF
2017-08-23T13:21:12+02:00 INFO Error publishing events (retrying): EOF
2017-08-23T13:21:27+02:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.publish.read_errors=5 libbeat.logstash.publish.write_bytes=2419 libbeat.logstash.published_but_not_acked_events=10230 libbeat.publisher.published_events=2046
2017-08-23T13:21:28+02:00 ERR Failed to publish events caused by: EOF
2017-08-23T13:21:28+02:00 INFO Error publishing events (retrying): EOF
2017-08-23T13:21:57+02:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=282 libbeat.logstash.published_but_not_acked_events=2046
Here's filebeat.yml:
filebeat:
prospectors:
-
paths:
- /var/log/filebeat/logs/*.log
input_type: log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["localhost:5000"]
here's logstash configurations:
input {
beats {
port => 5044
}
}
output {
if "_grokparsefailure" not in [tags]{
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "elasticsearch:9200"
}
}
}
Can you help?