Drop Lines before multiline codec

Elastic Stack Logstash
1

I am parsing multiple files using the multiline codec .
Using the following to convert the file contents into one big string and then extract patterns i need from that string.

input {
    file {
            path => "/logstash_test/*.*"
            start_position => "beginning"
            codec => multiline {
                      pattern => "^ENDOFFILE$"
                      negate => "true"
                      what => "previous"
                    }
            sincedb_path => "/dev/null"
     }
}
filter{
some groks which works on the above big line .
}

  • Is there any way to drop lines from the log before of during the multiline codec is running . ie . if i have a file containing following text .

    [1;30m[14:52:49] e[0m e[0;35mCOMMAND e[0mPOST "/wd/hub/session"
    [1;30m[14:52:49] e[0m e[0;35mCOMMAND e[0mPOST "/wd/hub/session"
    [1;30m[14:52:49] e[0m e[0;33mDATA e[0m{"desiredCapabilities": {"javascriptEnabled":true,"locationContextEnabled":true,"handlesAlerts":true,"rotatable":true,"build":"roboradar-trial","browserstack.local":true,"os":"Windows","os_version":"7","browser":"Chrome","browser_version":32,"_":"4 more keys: ["name","browserName","loggingPrefs","requestOrigins"]"}}
    e[1;30m[14:52:49] e[0m e[0;33mDATA e[0m{"desiredCapabilities": {"javascriptEnabled":true,"locationContextEnabled":true,"handlesAlerts":true,"rotatable":true,"build":"roboradar-
    ENDOFFILE

How can i omit all lines starting with e[1;30m[14:52:49] while also merging the remaining lines using multiline. i need to negate some lines and retain others during multiline processing.

2

I don't think there's a way to do that, sorry.

3

Can i create a custom codec plugin that does this using the following

I was looking into

Could you please provide some guidelines regarding regular expression to use ?

4

Sure, making a custom codec is always possible. You could clone the multiline codec and just add a few lines of code to skip the unwanted lines.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.