# Drop old messages using Ruby filter
**URL:** https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990
**Category:** Logstash
**Created:** [October 11, 2018, 6:27am UTC](https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990 "2018-10-11T06:27:21Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@programagor](https://discuss.elastic.co/u/programagor)
#### Post date: [October 11, 2018, 6:27am UTC](https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990/1 "2018-10-11T06:27:21Z")
Greetings
Occasionally, my logstash receives a message from few days back, and logstash tries to write it into an index which was already marked as readonly and forcemerged. Logstash then receives a 403 error from Elasticsearch, and instead of dropping the message or placing it into DLQ, it keeps retrying it.
Eventually, these undeliverable messages clog up the output queue completely, and no valid events are emitted afterwards.
I submitted a bug report already, but there seems to be no activity on it:
>
>
> When Logstash encounters the 403 error from Elasticsearch, it erroneously reatte…mpts to index the document. This document then keeps polluting the output queue, potentially reducing throughput of the entire pipeline.
> The correct behaviour is to either place the document into DLQ, or to drop the document entirely.
>
> From RFC 2616 - 10.4.4 403 Forbidden:
> \> The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated.
>
> I encountered this problem when I set older indices to be read-only, and Logstash picked up some old logs and tried to write them into these old read-only indices.
> These messages are logged:
> \`\`\`
> \[2018-09-26T16:20:04,391\]\[INFO \]\[logstash.outputs.elasticsearch\] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: \[FORBIDDEN/8/index write (api)\];"})
> \[2018-09-26T16:20:04,391\]\[INFO \]\[logstash.outputs.elasticsearch\] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>4}
> \`\`\`
>
> \- Version: 6.4.1
> \- Operating System: CentOS Linux release 7.4.1708 (Core), GNU/Linux 4.15.0-29-generic x86\_64
> \- Config File (if you have sensitive info, please remove it)
> \`\`\`
> node.name: xxxxx
> path.data: /var/lib/logstash
> path.logs: /opt/logstash/logs
> http.host: x.x.x.x
> http.port: 9600-9700
> xpack.monitoring.enabled: true
> xpack.monitoring.elasticsearch.url: \["https://x.x.x.x:9200", ...\]
>
> xpack.monitoring.elasticsearch.username: "logstash\_system"
> xpack.monitoring.elasticsearch.password: "xxxxx"
> xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/xxxxx-ca.crt
>
> xpack.management.enabled: true
> xpack.management.elasticsearch.url: \["https://x.x.x.x:9200", ...\]
> xpack.management.elasticsearch.username: "logstash\_admin"
> xpack.management.elasticsearch.password: "xxxxx"
> xpack.management.logstash.poll\_interval: 5s
> xpack.management.pipeline.id: \["xxxxx"\]
> xpack.management.elasticsearch.ssl.ca: /etc/logstash/certs/xxxxx-ca.crt
> dead\_letter\_queue.enable: true
> \`\`\`
> \- Steps to Reproduce:
> 1. Create index
> 1. Set \`index\_settings.index.blocks.write: True\`
> 1. Write to the index using Logstash
>
> Associated discussion: https://discuss.elastic.co/t/make-logstash-drop-documents-on-403/149977
I tried using the following filter:
```auto
filter {
ruby {
init => "require 'time'"
code => 'if event.get("[@timestamp]") < ( Time.now - 432000 )
event.cancel
end'
}
}
```
However, using this, I get the following error:
```auto
[ERROR][logstash.filters.ruby] Ruby exception occurred: comparison of LogStash::Timestamp with Time failed
```
I also tried to follow this answer, but in my environment, installing additional plugins like `age{}` is a lengthy process requiring aprovals, so I'd prefer to avoid that:
> [@Drop old messages](https://discuss.elastic.co/t/drop-old-messages/111775):
>
> Hi all, i'm reading with logstash several files and after send to ES. I this files i have also old log lines with timestamps. How can avoid to send to ES the rows with old timestamp? I can use drop filter, but what is the condition to check? BR
Is there an easy way to use the Ruby filter to achieve this?
Thanks
---
### Author:  [@programagor](https://discuss.elastic.co/u/programagor)
#### Post date: [October 19, 2018, 7:51am UTC](https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990/2 "2018-10-19T07:51:47Z")
I found a way:
```auto
ruby {
init => "require 'time'"
code => 'if LogStash::Timestamp.new(event.get("@timestamp")+432000) < ( LogStash::Timestamp.now)
event.cancel
end'
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 16, 2018, 7:51am UTC](https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990/3 "2018-11-16T07:51:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.