Delete all extracted events before a specific field

Elastic Stack Logstash
1

Hi there, I am having an important message in my logfile (logfile stats, comes at the end of the logfile).
Sometimes the logfile is not finished (before sending it to logstash), so the stats message is not written yet, so in this case I parse the stats out of the logfile line by line. However, if the message is found (logfile is finished), I won't need all the extracted events from above, I only need to parse this message, and drop all extracted and saved events before this message, how can I do this?

  • I am sending my output to elasticsearch
2

How are you sending the logs? I mean,what is your input configuration in Logstash? Are you reading from a file?

The message that you want has any unique identifier, for example a field that only this line message will have?

3

Hey @leandrojmp, thanks for replying, I am sending the logs using TCP, yes my message is unique, what I am asking is how to delete all the events before this message in case it's found?

4

For what I understand, each line in your log files you run through the pipeline, I do not know if it is possible to delete an event that already passed through the pipeline (input-filter-output) based on a later event.

However, you can drop events that do not match a conditional, maybe something like this.

filter {
    if "unique-id" not in [message] {
        drop { }
    }
}

But this way you will drop all the events that do not have the unique-id, if you have to keep those events you could separate using tags or redirecting to other output then you would have an output for the lines that have the full stats you want and an output for the other lines.

I will try to replicate your problem later and see if I can come back with another solution, it's an interesting use case.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.