# Stop exporting logs after file reaches a certain size **URL:** \ **Category:** Logstash\ **Created:** [March 18, 2021, 7:21pm UTC](https://discuss.elastic.co/t/stop-exporting-logs-after-file-reaches-a-certain-size/267712 "2021-03-18T19:21:20Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![agushchin](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@agushchin](https://discuss.elastic.co/u/agushchin)\ **Post date:** [March 18, 2021, 7:21pm UTC](https://discuss.elastic.co/t/stop-exporting-logs-after-file-reaches-a-certain-size/267712/1 "2021-03-18T19:21:20Z")
Hi, I have the following question/problem: is there a way to stop logstash sending logs to ES after the log file reaches a certain size limit (or some other limit is reached)? Asking because recently faced an issue caused by a bug in app code that generated a huge number of duplicated/useless log messages and the log file reached 100G limit. I understand that the issue in the app must be fixed in the first place, but still curious. Thanks in advance. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 18, 2021, 8:06pm UTC](https://discuss.elastic.co/t/stop-exporting-logs-after-file-reaches-a-certain-size/267712/2 "2021-03-18T20:06:04Z")
In logstash you could use a [throttle](https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html) filter to tag events and then a drop filter to delete them. This would, for example, allow you to restrict logstash to outputting 1,000 events per minute. In elasticsearch you can use ILM to rollover indexes based on size. I think you can limit the total size using curator by having a [count](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/filtertype_count.html) filter on a delete action. I am not aware of a way to do count based deletion in ILM, but I do not run elasticsearch so I may just have missed it. You might be better asking in the elasticsearch forum. ---
**Author:** ![agushchin](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@agushchin](https://discuss.elastic.co/u/agushchin)\ **Post date:** [March 18, 2021, 11:18pm UTC](https://discuss.elastic.co/t/stop-exporting-logs-after-file-reaches-a-certain-size/267712/3 "2021-03-18T23:18:39Z")
Thanks Badger, I will try using throttle. My understanding (please correct if I am wrong) is that I can limit the number of identical log messages per unit of time from the same host by using `key => "%{host}%{message}"`. I am actually using ILM based on the index size, but it doesn't prevent from appearing a lot of identical useless log messages. Also AFAIK ILM can only roll/delete the whole index, whereas I would prefer just to get rid of these specific messages. I also asked a similar question in filebeat forum to check if filebeat could handle this situation with huge files/repeated log messages, but didn't get any response there. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 18, 2021, 11:26pm UTC](https://discuss.elastic.co/t/stop-exporting-logs-after-file-reaches-a-certain-size/267712/4 "2021-03-18T23:26:28Z")
> [@agushchin](#): > > My understanding (please correct if I am wrong) is that I can limit the number of identical log messages per unit of time from the same host by using `key => "%{host}%{message}"` . Yes, if the [message] field is actually identical then you can use that. The example in the documentation does exactly that. For completeness, another option would be to set the document\_id option on the elasticsearch output as a hash of the [host] and [message] fields (using a fingerprint filter), so that multiple copies of the same message will overwrite oneanother. I cannot see any reason to prefer that to a throttle filter though. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 15, 2021, 11:27pm UTC](https://discuss.elastic.co/t/stop-exporting-logs-after-file-reaches-a-certain-size/267712/5 "2021-04-15T23:27:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.