Hi, I have the following question/problem: is there a way to stop logstash sending logs to ES after the log file reaches a certain size limit (or some other limit is reached)? Asking because recently faced an issue caused by a bug in app code that generated a huge number of duplicated/useless log messages and the log file reached 100G limit. I understand that the issue in the app must be fixed in the first place, but still curious. Thanks in advance.
In logstash you could use a throttle filter to tag events and then a drop filter to delete them. This would, for example, allow you to restrict logstash to outputting 1,000 events per minute.
In elasticsearch you can use ILM to rollover indexes based on size. I think you can limit the total size using curator by having a count filter on a delete action. I am not aware of a way to do count based deletion in ILM, but I do not run elasticsearch so I may just have missed it. You might be better asking in the elasticsearch forum.
Thanks Badger, I will try using throttle. My understanding (please correct if I am wrong) is that I can limit the number of identical log messages per unit of time from the same host by using key => "%{host}%{message}".
I am actually using ILM based on the index size, but it doesn't prevent from appearing a lot of identical useless log messages. Also AFAIK ILM can only roll/delete the whole index, whereas I would prefer just to get rid of these specific messages.
I also asked a similar question in filebeat forum to check if filebeat could handle this situation with huge files/repeated log messages, but didn't get any response there.
Yes, if the [message] field is actually identical then you can use that. The example in the documentation does exactly that.
For completeness, another option would be to set the document_id option on the elasticsearch output as a hash of the [host] and [message] fields (using a fingerprint filter), so that multiple copies of the same message will overwrite oneanother. I cannot see any reason to prefer that to a throttle filter though.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.