# Logstash S3 input plugin - filter based on time modified **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [December 21, 2021, 3:50pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581 "2021-12-21T15:50:31Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![kmualem](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@kmualem](https://discuss.elastic.co/u/kmualem)\ **Post date:** [December 21, 2021, 3:50pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/1 "2021-12-21T15:50:31Z")
I have a Logstash container that is configured to read objects from S3. The requirement is to filter old objects, let's say objects before 3 months should be dropped. I noticed that I can expose the s3 metadata, so I have the following metadata in each event: ```auto "@metadata" => { "s3" => { "etag" => "\"xxx"", "content_length" => 33, "metadata" => {}, "version_id" => "null", "accept_ranges" => "bytes", "last_modified" => 2021-12-21T13:30:28.000Z, ``` Maybe there is a filter/ruby code that I can use in order to filter "old" objects and drop them? Any help is appreciated! ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [December 21, 2021, 4:47pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/2 "2021-12-21T16:47:09Z")
There is a great newer filter called "[age](https://www.elastic.co/guide/en/logstash/current/plugins-filters-age.html)" which may work for your use case. Just be sure to update your `@timestamp` with the `last_modified` field using the [date](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html) filter first and then run the age filter with your conditional drop statement. ```auto date { match => ["[@metadata][s3][last_modified]", "ISO8601" ] target => "@timestamp" } age {} #One month = 2629746 #Three months = 7889238 if [@metadata][age] > 7889238 { drop {} } ``` Let me know if this works for you. ---
**Author:** ![kmualem](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@kmualem](https://discuss.elastic.co/u/kmualem)\ **Post date:** [December 21, 2021, 5:14pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/3 "2021-12-21T17:14:58Z")
Thanks a lot! @AquaX I was familiar with the "age" plugin, but actually, I didn't know if this is a good approach to override the original timestamp value. First, I will make a try and check your suggestion and also verify if I have some limitations to override the timestamp value. I will update... Thanks! ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [December 21, 2021, 5:17pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/4 "2021-12-21T17:17:13Z")
Overriding the `@timestamp` field is a common practice and will allow you to actually properly see the data in a timeline from when it was originally generated. If you want you can make a duplicate of the `@timestamp` field first and save it in another field so you can capture the "processed" time then that's another thing you could do (I do this in my environment). Let me know if this solves your issue. ---
**Author:** ![kmualem](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@kmualem](https://discuss.elastic.co/u/kmualem)\ **Post date:** [December 21, 2021, 5:50pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/5 "2021-12-21T17:50:02Z")
I agree I sound very reasonable. In the first place, I thought that I might change some existing logic that uses the timestamp... but I think that this is fine. So, I had to install the plugin as it was not installed in the image that I'm using. I started the approach of first parsing the field with the date filter to aother field, in order to see that the parsing are fine, but it seems like I have a problem. First, should I use the exact path of the last\_modified field? something like ```auto date { match => ["[@metadata][s3][last_modified]", "ISO8601" ] target => "s3Time" } ``` No? Second, it seems like this is failed to parse, I see in the logs: "tags" =\> [ [0] "\_dateparsefailure" ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [December 21, 2021, 6:11pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/6 "2021-12-21T18:11:50Z")
I see it! Looking at your data again it looks like `last_modified` is already being recognized as a timestamp type of field as there are no " " around the value. That's good! You can just rename or copy the field then using a mutate filter. No need for the date filter at all 😃 ```auto mutate { copy => { "[@metadata][s3][last_modified]" => "s3time"} copy => { "@timestamp" => "processed_time"} copy => {"[@metadata][s3][last_modified]" => "@timestamp"} } ``` Not sure if this code will work exactly as I don't have a logstash instance in front of me... play around with it. ---
**Author:** ![kmualem](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@kmualem](https://discuss.elastic.co/u/kmualem)\ **Post date:** [December 21, 2021, 6:28pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/7 "2021-12-21T18:28:57Z")
Great, sure! I will play with it... Thank you ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [December 21, 2021, 6:32pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/8 "2021-12-21T18:32:28Z")
Great! Please let me know if this solves your issue or mark this post as such 🙂 ---
**Author:** ![kmualem](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@kmualem](https://discuss.elastic.co/u/kmualem)\ **Post date:** [December 22, 2021, 7:39am UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/9 "2021-12-22T07:39:06Z")
Ok, so the direction that you gave me @AquaX was great. After some playing with the logstach and debugging I manage to configure the age plugin using the S3 last\_modified - In the mutate, in order to override the @timestamp I used copy as follows: ```auto copy => { "[@metadata][s3][last_modified]" => "@timestamp"} ``` Then, it seems like the age plugin make the correct filtering based on the S3 object time @AquaX feel free to edit you last comment and I will mark as you solved it 🙂 Thanks ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [December 22, 2021, 2:03pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/10 "2021-12-22T14:03:27Z")
Perfect! Glad you got it figured out. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 19, 2022, 2:03pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-filter-based-on-time-modified/292581/11 "2022-01-19T14:03:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.