Filtering rows from a log based in their last status

ManuelChaverra · May 15, 2020, 5:14pm

Hello everyone, this is my first time using Elastic Stack so I'm trying to learn a lot of things that I'd like to implement in my work.

I'm using Filebeat to send logs files to logstash, then I'm filtering information and sending it to elasticsearch to finally be able to review it in Kibana (So Filebeat > Logstash > Elasticsearch > Kibana)

But I'm have the following problem:
My log file looks like this:

I need to extract the rows corresponding to the last status of each process, for example, if I have the following rows:

I need to return the following since they contain the last status (based on the date) of the process (RPA202 and RPA313 respectively) and ran in the same machine (Maquina_18 and Maquina_5 respectively):

So, to achieve this, I tried a dissect filter in the logstash config file to indicate the pipes as separators and then use Kibana filters to get the information as I need it, but I haven't achieve it.

My configuration file looks like this:

I would appreciate a lot any advice or hint to achieve it

Thanks in advance! I hope I make myself clear because English is not my first language, so I'm sorry If I had any mistake writing this.

Badger · May 15, 2020, 9:44pm

If you want to do it in logstash (rather than by using an elasticsearch query) then you could use an aggregate filter to determine the last message. See example 3.

Make sure you set pipeline.workers to 1 and disable java_execution, otherwise events get processed out of order and last is no longer last.

system · June 12, 2020, 9:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.