Logstash - complicated parsing, can logstash remember states?

[yoav1 (yoav)]

Hi,

I am new in elasticsearch and logstash.

my logstash receives its input from filebeat and parses the file line by line.

I am trying to do compilcated parsing and wonders if logstash can remember things that seen in previous ines.

for example:

line 1: "this is line 1 the value for group 1 id is [abcd]
line 2: "this is line 2 the value for group 2 id is [efgh]"
line 3: "[efgh] this is line 3"
line 4: "this is line 3 the value for group 4 id is [ijkl]"
line 5: "[efgh] this is line 5"
line 6: "[abcd] this is line 6"
line 7: "[ijkl] this is line 7"

I would like logstash to search the group id's that appears after the string "the value for group x id is"
remember them and for each line insert to elasticsearch its group number as value of "group number" key

how can I do this?

anothe example:
line 1: "this is start of paragraph 1"
line 2: "this is line 2"
line 3: "this is line 3"
line 4: "this is end of paragraph 1"
line 5: "this is start of paragraph 2"
line 6: "this is line 6"
line 7: "this is line 7"
line 8: "this is start of paragraph 2"

I would like logstash to insert for each line the paragraph it belongs to
in this example - lines 1,2,3,4 - belongs to paragraph 1
lines 5678 - belongs to paragraph 2

how can I do it?

[Makra (Makara)]

I guess, you can do it using a ruby filter, where you scan for start of the paragraph and go on adding lines to a string until end of paragraph is reached. After that you can add the string(combined lines) to the message field and drop the individual lines if you desired.

By the way, What is the final message filed you expect ?

[yoav1 (yoav)]

Thanks
The files can be very big so i guess that adding the lines to one string will not be the best practice.

The final message should be the line itself
For example :
Message: this is line 3
Paragraph: 1

Do you have an idea on how to do it?