elasticsearch is running on a 3 node cluster, logstash and filebeat are running on a single node
all version are: 6.5.1
logs are being sent to elasticsearch for short periods (~2 hours), then they stop... only after restart logstash they resume
Can you share logstash logs , are there any event rejections in logs
lines of log since restart:
[2019-03-31T09:55:18,788][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-31T09:55:19,236][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2049, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,202][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,313][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,313][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,312][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,313][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,313][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,313][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,333][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,333][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,333][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,333][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,333][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,334][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,345][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,345][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,345][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,346][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,346][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,346][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-03-31T09:55:20,350][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 260 from source id 2065, because no template to decode it with has been received. This message will usually go away after 1 minute.
...
[2019-03-31T10:03:12,881][INFO ][logstash.outputs.file ] Opening file {:path=>"/logstash/parse_failure/trustwave_waf.log"}
[2019-03-31T10:04:52,067][INFO ][logstash.outputs.file ] Closing file /logstash/parse_failure/trustwave_waf.log
...
I can move into debug mode...
No need to move into debug, issue with filebeat that caused logstash to get strangled with worker threads...
filebeat was including logs for .log.* so, every time a new GZIP'd file was created it opened a new pipeline on logstash... logstash top was extremely high due to that
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.