Logstash Periodically Failing

Elastic Stack Logstash
1

Every few days I notice my logstash stops ingesting data. When I check the log I find this:

[2021-06-29T03:30:39,236][ERROR][logstash.javapipeline ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"Counters can not be incremented by negative values",

I usually end up restarting all three services (ELK) and it starts working again. Can someone point me in the right direction for troubleshooting?

2

The exception is happening here. This issue suggests it may be caused by NTP moving time backwards (but make sure you follow the links to see why folks think that is not so).

3

Thank you.

Hmm...it looks like the original issue was resolved in 6.0.1 and 6.1.0. I am running 7.10.1. Although I see the response on this post found a work around with his filter. I do not have a metrics filter like that user though. So I'm still not sure where to go from here.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.