Issue for the multiline input

Elastic Stack Logstash
1

Hi
@leandrojmp There we will continue topic from Grok for data - #13 by leandrojmp
now we need to complete process for parsing below date with the exact content:

input {
    file {
        mode => read
        path =>"/opt/data/input/export_a_202207*"
        codec => multiline {
                                pattern => "#"
                                what => previous
                                negate => true
                                max_lines => 600000
                                max_bytes => 20000000
                                auto_flush_interval => 5
                                }
        sincedb_path => "/dev/null"
    }
}

filter {
    mutate {
        gsub => ["message", "\n",";"]
    }
    mutate {
        split => { 
            "message" => ";"
        }
    }
    dissect {
        mapping => {
            "[message][0]" => "# %{activity},%{val},%{time}"
        }
        remove_field => ["[message][0]"]
    }
    split {
        field => "message"
    }
    date {
        match => ["time", "yyyyMMddHHmmss"]
        timezone => "Europe/Paris"
    }
    dissect {
        mapping => {
            "message" => "%{}(%{type} %{numvalue}, %{status} %{statusval}, %{list} %{listval})"
        }
    }
} #filter


output {

   stdout { codec => rubydebug{}

                }
        }

sample data:
[file1](Easyupload.io - Upload files for free and transfer big files easily.)
data highlight:

# snapshot,66593881,20220711061504
list_of_count(number 00000801163158, abort 0, onlist yes)
list_of_count(number 00001004020051, abort 0, onlist yes)
list_of_count(number 00006053206182, abort 0, onlist yes)
list_of_count(number 00006053303925, abort 0, onlist yes)
list_of_count(number 00007504958630, abort 0, onlist yes)
list_of_count(number 00009053373401, abort 0, onlist yes)
list_of_count(number 00060633330647, abort 0, onlist yes)
list_of_count(number 00090533747262, abort 0, onlist yes)
list_of_count(number 00114798160038, abort 0, onlist yes)
# 20220711061537

[file2](Easyupload.io - Upload files for free and transfer big files easily.)

2

Did you test it to see if you need to change anything?

Both of your files only have one pair of header and footer starting with #, if this is the case, then you don't need to change anything in the multiline as it will work as expected, maybe just increase the auto_flush_interval.

3

I've decided to test it without auto_flush_interval (as I understood auto_flush_interval is required because otherwise it will wait forever for a line that does match pattern => "#")?

  • in the second step I've tried to increase auto_flush_interval but got the same results.

at least in the logs I can find:


[WARN ] 2022-07-13 17:54:47.915 [[test]>worker0] Dissector - Dissector mapping, pattern not found {"field"=>"[message][0]", "pattern"=>"# %{activity},%{val},%{time}", "event"=>{"path"=>"/opt/data/input/test_20220713061502.txt", "host"=>"0.0.0.0", "@timestamp"=>2022-07-13T17:54:47.871711Z, "tags"=>["_dissectfailure"], "@version"=>"1", "message"=>["# 20220711061537"]}}
[WARN ] 2022-07-13 17:54:47.916 [[test]>worker0] Dissector - Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{}(%{type} %{numvalue}, %{status} %{statusval}, %{list} %{listval})", "event"=>{"path"=>"/opt/data/input/test_20220713061502.txt", "host"=>"0.0.0.0", "@timestamp"=>2022-07-13T17:54:47.871711Z, "tags"=>["_dissectfailure"], "@version"=>"1", "message"=>"# 20220711061537"}}
{
4

As I thought it should work after add one more dissect:

 dissect {
        mapping => {
            "message" => "# %{time}"
        }
                remove_field => ["message"]
    }
5

Do you want to use this information in your documents? If not, you do not need this dissect and can ignore the errors, this is being parsed as a different event from the others.

If you want this information to be part of the same event then I think you will need to change the way you ingest your files, maybe using the file input in the read mode since each file seems to have just the information regarding one execution of this tool you are using.

6

Ok, I will remove this last line from files through simple bash script:

head -n -1 foo.txt | sponge foo.txt

Many thanks for help!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.