Processing large amoung of data with Logstash

Elastic Stack Logstash
5

I agree with Jenni. Use dissect to parse the first part of the line then use ruby. You could use the .scan method of the String class

    ruby {
        code => '
            s = event.get("tests_string")
            if s
                event.set("matches", s.scan(/\s*([^;]+); ([^;]+); ([^;]+)(;|$)/))
            end
        '
    }

which will result in a variable length array such as

       "matches" => [
    [0] [
        [0] "Test1",
        [1] "Result1",
        [2] "Comment1",
        [3] ";"
    ],
    [1] [
        [0] "Test2",
        [1] "Result2",
        [2] "Comment2",
        [3] ""
    ]
],

You will likely want to iterate over the array and reformat the data.

1 Like