# Delete all extracted events before a specific field

**URL:** <https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847>\
**Category:** Logstash\
**Created:** [July 27, 2017, 6:11pm UTC](https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847 "2017-07-27T18:11:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [July 27, 2017, 6:11pm UTC](https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847/1 "2017-07-27T18:11:04Z")

</div>

Hi there, I am having an important message in my logfile (logfile stats, comes at the end of the logfile).  
Sometimes the logfile is not finished (before sending it to logstash), so the stats message is not written yet, so in this case I parse the stats out of the logfile line by line. However, if the message is found (logfile is finished), I won't need all the extracted events from above, I only need to parse this message, and drop all extracted and saved events before this message, how can I do this?

- I am sending my output to elasticsearch

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 27, 2017, 9:02pm UTC](https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847/2 "2017-07-27T21:02:22Z")

</div>

How are you sending the logs? I mean,what is your input configuration in Logstash? Are you reading from a file?

The message that you want has any unique identifier, for example a field that only this line message will have?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [July 28, 2017, 1:09pm UTC](https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847/3 "2017-07-28T13:09:48Z")

</div>

Hey @leandrojmp, thanks for replying, I am sending the logs using TCP, yes my message is unique, what I am asking is how to delete all the events before this message in case it's found?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 28, 2017, 3:05pm UTC](https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847/4 "2017-07-28T15:05:59Z")

</div>

For what I understand, each line in your log files you run through the pipeline, I do not know if it is possible to delete an event that already passed through the pipeline (input-filter-output) based on a later event.

However, you can drop events that do not match a conditional, maybe something like this.

```auto
filter {
    if "unique-id" not in [message] {
        drop { }
    }
}

```

But this way you will drop all the events that do not have the unique-id, if you have to keep those events you could separate using tags or redirecting to other output then you would have an output for the lines that have the full stats you want and an output for the other lines.

I will try to replicate your problem later and see if I can come back with another solution, it's an interesting use case.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 25, 2017, 3:06pm UTC](https://discuss.elastic.co/t/delete-all-extracted-events-before-a-specific-field/94847/5 "2017-08-25T15:06:06Z")

</div>

Closed on August 25, 2017, 3:06pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.