Logstashs event parsing

Elastic Stack Logstash
1

Hello,

i am new to ELK. when i onboarded the below log file, it is going to "dead letter queue" in logstash because logstash couldn't able to process the events. I am not sure which type of plugin to use whether KV plugin or CSV plugin because first half of the events are normal and next half is in KV pair. Any help would be appreciated on how to write the filters.

Below is the sample log format.

25193662345 [http-nio-8080-exec-44] DEBUG c.s.b.a.m.PerformanceMetricsFilter - method=PUT status=201 appLogicTime=1, streamInTime=0, blobStorageTime=31, totalTime=33 tenantId=b9sdfs-1033-4444-aba5-csdfsdfsf, immutableBlobId=bss_c_586331/Sample_app12-sdas-157123148464.txt, blobSize=2862, domain=abc

2519366789 [http-nio-8080-exec-47] DEBUG q.s.b.y.m.PerformanceMetricsFilter - method=PUT status=201 appLogicTime=1, streamInTime=0, blobStorageTime=32, totalTime=33 tenantId=b0csdfsd-1066-4444-adf4-ce7bsdfssdf, immutableBlobId=bss_c_586334/Sample_app15-615223-157sadas6648465.txt, blobSize=2862, domain=cde

Thanks

2

Try using a dissect filter to parse the first part of the line, and a kv filter for the rest.

    dissect { mapping => { "message" => "%{someNumber} [%{thread}] %{level} %{class} - %{[@metadata][restOfLine]}" } }
    kv { source => "[@metadata][restOfLine]" field_split => "," }
3

Thank you for the reply. i have changed the filter as per the suggestion. But when i run it in the GROK debugger i am getting compiler error. Below is the filter which i have used.

dissect { mapping => { "message" => "%{NUMBER:number} [%{thread}] %{level} %{class} - %{[@metadata][msg]}" } }
    kv {
              source => "[@metadata][msg]"
              include_keys => [ "method", "status", "appLogicTime", "streamInTime", "blobStorageTime", "totalTime", "tenantId", "immutableBlobId", "blobSize", "domain" ]
              field_split_pattern => "\,?\s"
              value_split => "="
       }
4

You cannot use a grok debugger to test dissect or kv filters. It is specific to the grok filter.