I was using logstash to forward filebeat info (zeek and suricata) to the elastic cluster I manage. Filebeat is 7.12.0 and logstash and elastic are version 7.13.4. It was working without a problem until yesterday.
I am now sending the data straight from the network sensor that filebeats is on into our stack and that is working fine. I am just confused as I can't find any reason why the logs suddenly stopped going through logstash.
I wasn't getting any error on logstash or filebeat to indicate that anything was wrong. We noticed that running the following: curl -XGET 'localhost:9600/_node/stats/pipelines' showed that events weren't coming into or leaving logstash. The in and out numbers remained the same no matter how many times we executed.
I disabled the persistent queue on the logstash and ran the curl command a few times and it was clear that events were now making it through the logstash, but the filebeats weren't making it into the stack.
The issue here is that I use one pipeline for processing winlogbeat data from endpoints as well as the filebeat data from network sensors. We just use a simple if statement in the logstash output to get them into the right indices. So, I don't know if those numbers are just the winlogbeat data making it through successfully or what is going on.
Have you done any recent upgrades to anything?
Are there any errors in the logstash or operating system logs? If you're using persistent queues then has the drive where the queues are on filled up?
No errors in logstash. What OS logs would you suggest checking?
As far as the persistent queues are concerned before I commented them out the drive didn't fill up, but it reached the limit that I had it configured to. At which point I deleted the data in the persistent queue. I then increased the limit for the persistent queue. I started logstash then watched the persistent queue fill up again to the new limit. At the same time I used: curl -XGET 'localhost:9600/_node/stats/pipelines' to monitor if logstash was processing events. It didn't look like it was.
Then I commented out the persistent queue, and according to the curl command logstash was filtering events (if I read it/used it correctly). Though the events didn't make it to Elasticsearch.
The final move was to change the output on filebeat from logstash to the elastic stack. Now the events are populating in the stack successfully, so I am trying to loop back and see what I set up incorrectly on logstash. I have just run out of diagnosis tools that I am aware of to find evidence of trouble.
Thanks for the details.
It's hard to know what is or is not going on. My recommendation would be to use the simplest logstash configuration possible (simple input/stdout or file output) and see if the events are making it through that.
Then you can start adding in different components to see when it breaks.
My guess is that there is still something wrong with the persistent queue settings or your filter that is causing the events to be dropped.
Check that your persistent queues are not corrupted and you may also want to setup a dead letter queue to capture anything that is not properly processed.
We have a very simple config file. I can share if that would helpful. We are not filtering anything. Just using a single pipeline to output the information to a winlogbeat or filebeat index depending on the presence of filebeat data.
As far as the persistent queue settings we have six logstashes with the same configuration and suddenly one of them stopped forwarding data correctly. The only thing that I changed was that I commented out the persistent queue and enabled the dead letter queue to see what was failing. When I did that nothing went to the dead letter queue so it seems like everything was being processed properly.
When the persistent queues were enabled it seems like they are working. They just continuously filled up until they hit the max configured limit.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.