Extract multiline log with not id field present

Elastic Stack Logstash
#1

Hello.

I need to build a record composed of multiple lines where there is no clear identifier

  • There is a word that tells me where the task starts in this case "select"
  • There is a word that tells me where the task ends in this case "end log"
  • I don´t have a field that allows me to uniquely identify
  • Important information exists on various lines
  • everything else must be ignored

I have a log file with similar estructure:

    05 Feb 2021 14:00:00,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:00:00,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:00:00,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:00:01,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:00:01,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:00:01,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:00:02,213 [AAA-11] INFO  - select name

    05 Feb 2021 14:00:02,213 [AAA-11] INFO  - Include important info

    05 Feb 2021 14:00:02,231 [AAA-11] INFO  - message

    <asd>

        <tag1>Value 1</tag1>

        <tag2>Value 2</tag2>

    </asd>

    05 Feb 2021 14:00:03,131 [AAA-11] INFO  - Include other important info

    05 Feb 2021 14:00:03,131 [AAA-11] INFO  - end log

    05 Feb 2021 14:00:03,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:00:03,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:00:03,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:00:04,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:00:04,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:00:04,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:00:05,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:00:05,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:00:05,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:01:00,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:01:00,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:01:00,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:01:01,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:01:01,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:01:01,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:01:02,213 [AAA-11] INFO  - select name

    05 Feb 2021 14:01:02,213 [AAA-11] INFO  - Include inportant inf

    05 Feb 2021 14:01:02,231 [AAA-11] INFO  - message

    <asd>

        <tag1>Value 1</tag1>

        <tag2>Value 2</tag2>

    </asd>

    05 Feb 2021 14:01:03,131 [AAA-11] INFO  - Include other important info

    05 Feb 2021 14:01:03,131 [AAA-11] INFO  - end log

    05 Feb 2021 14:01:03,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:01:03,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:01:03,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:01:04,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:01:04,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:01:04,231 [AAA-11] INFO  - Line3[: 0]

    05 Feb 2021 14:01:05,213 [AAA-11] INFO  - Line A:[0]

    05 Feb 2021 14:01:05,231 [AAA-11] INFO  - Line2:[0]

    05 Feb 2021 14:01:05,231 [AAA-11] INFO  - Line3[: 0]
#2

If you are using a file input you could use a multiline codec to roll up lines until an "end log" is seen. You could then use mutate+gsub to remove everything before "select". Then pick the resulting event apart with grok, perhaps.

#3

Logstash insert documents with tag:

`"multiline_codec_max_lines_reached"`

    input {

        file {

            path => ["./my.log"]

            start_position => "beginning"

            sincedb_path => "NUL"

            codec => multiline {

                pattern => "^end log"

                what => "previous"

                negate => true

            }

        }

    }

    filter {

        mutate {

            gsub => ["message", '\r', ""]

        }

    }

    output {

        elasticsearch {

            hosts => ["http://127.0.0.1:9200"]

            index => "send"

            document_type => "_doc"

        }

    }
#4

That is anchored to start of line. If end log is not at the start of the line that will roll up the entire file into a single event.

#5

For this case the word "select" init the event and the "end log" in other line finish the event. I would have to capture the lines that are in between.

#6

Remove the ^ from your pattern option.

#7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.