In pipeline I receive logs from server with elasticsearch input plugin and I want to process not all logs, but for example only last 2 months, or, preferably from certain date to certain date. When I receive logs there is already separate field @timestamp, so I tried to use this input { elast…

[image] John_Smith1: if 86400 < [@metadata][age] { # grok and other conditions. The events will still go through the pipeline unless you drop them. That condition means event will only go through the grok filter if they are more than one day old. Perhaps what you want is if [@metadat…

Badger July 14, 2021, 6:21pm 2

There is an example of testing the age of an event here.

Topic		Replies	Views
Identify lines older than X days Logstash	12	3800	July 6, 2017
How to process old logs? Logstash	1	782	November 26, 2018
Elasticsearch output with a time period Logstash	1	398	November 27, 2017
Date filter plugin issue Logstash	13	387	August 1, 2018
Optimizing Datefield processing by moving from date{} to ruby code Logstash	1	276	February 17, 2020

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

How to process logs from certain period of time? (I use elasticsearch input plugin)

Related topics