Logstash Memorization

Elastic Stack Logstash
1

Hi everyone,

I need any help about Logstash. I have to add a common field to all lines. For example:

Monday [Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server
[Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server
[Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server
Tuesday [Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server
[Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server
[Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server

I would like to add "Monday" to one lines. Moreover, I would like "Tuesday" to the others lines.

I have done it with "Multiline" treating the "weekday" lines as a block, grok them to idenfity the "weekday" and after that split them to treat them separately. The problem is that the blocks I manage are too big and consume a lot of memory.

Is there any other way to do it?

Thanks in advance,

Rodrigo

2

How are you using the weekday name, i.e. why are you interested in keeping it as a separate field?

3

Hi, Magnus

I use it because I want to know which lines belong to each day of the week.

Regards

4

That doesn't really answer my question. You might be helped by https://qbox.io/blog/elasticsearch-aggregations-weekday-hourly-analysis or you should look into extracting the name of the weekday from the timestamp. Either use a ruby filter to do it on the fly or use a translate filter to look up the date part of the timestamp against a static list of date-to-weekday mappings.

5

This is a sample data. The goal is to grok a field and add it (memorized) to a determinated bounch of documents.

Is it possible?

Thanks in advance

6

This is a sample data.

Not a very good example then.

The goal is to grok a field and add it (memorized) to a determinated bounch of documents.

Not really, but the aggregate filter is probably the closest one.

7

I'll try to provide you a better example. Following lines are like the real ones:

11:36:59 [I] :=====================================================================
11:36:59 [I] : ========================= ------- BEGIN ------- =============================
11:36:59 [I] : Running process at : 06/09/2016 11:36:59
11:36:59 [I] : START 
11:36:59 [I] : Simulation
11:36:00 [W] : Trying to get configuration file
11:36:00 [W] : Opening configuration file : Ok
11:38:59 [I] : ========================= ------- END ------- ==============================
11:39:59 [I] : ===============================================================================
11:39:59 [I] : ========================= ------- BEGIN ------- =============================
11:39:59 [I] : Running process at : 17/09/2016 11:39:59
11:39:59 [I] : START 2
11:39:59 [I] : Simulation3
11:39:00 [W] : Trying to get configuration file
11:39:00 [W] : Opening configuration file : Ok
11:40:59 [I] : ========================= ------- END ------- ==============================

We need to get the Date at the "Running process at" and add the field to the all lines between the BEGIN and END it belongs. The problem with aggregate is that we have not a common taskid to group them.

Thanks in advance again,

Regards