Dissect failure for no obvious reason

Elastic Stack Logstash
1

Hi there,

For no obvious (to me) reason, a certain message coming from filebeat that always fell in error on logstash processing.
Here is the sample:

Oct  1 09:38:12 ff434-srv1 MC:  debug  {74308} [ApiUiJob] PlanJob()

Here is the error:

[2020-10-01T09:35:50,584][WARN ][org.logstash.dissect.Dissector][main][9ab8d72ff21e241be6563d89de5f0cf5848ec428093ac24552f7e2baca09cb77] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{ts->} %{+ts} %{+ts} %{Theater} %{Product->} %{Severity->} %{Cycle} %{Subject->} %{Body}", "event"=>{"log"=>{"offset"=>2302196, "file"=>{"path"=>"/var/log/wblog"}}, "@version"=>"1", "@timestamp"=>2020-10-01T09:35:49.446Z, "message"=>"Oct  1 09:35:48 ff434-srv1 MC:  debug  {63258} [ApiUiJob] PlanJob()", "tags"=>["beats_input_codec_plain_applied", "_dissectfailure"], "agent"=>{"name"=>"ff434-srv1", "id"=>"060bcbed-78c4-4c61-94d5-13131e0f0667", "hostname"=>"ff434-srv1", "version"=>"7.9.1", "type"=>"filebeat", "ephemeral_id"=>"23fbafb5-8f0e-48a8-bf7d-2c70e54e8c1b"}, "input"=>{"type"=>"log"}, "ecs"=>{"version"=>"1.5.0"}}}

If I inject the exact same log in the log file with echo, I don't get that result and the log succeed to passe dissect.
I also tried that config on that page https://dissect-tester.jorgelbg.me/ and everything is ok.

here is an extract of the config:

 if [log][file][path] == "/var/log/wblog" {
       dissect {
         mapping => {
           "message" => "%{ts->} %{+ts} %{+ts} %{Theater} %{Product->} %{Severity->} %{Cycle} %{Subject->} %{Body}"
         }
       }

       # extract the machine name
       grok {
         match => { "Subject" => "(?<Machine>\[([a-zA-Z0-9]{1,4}(:[a-zA-Z0-9]{1,4})?)\])" }
       }

       if [Machine] {
         # remove brackets from machine name
         mutate {
           gsub => [
             "Machine", "[\[\]]", ""
           ]
         }
       }

       kv { source => "Body" }

       mutate {
         uppercase => ["Severity"]
       }
     }

Nothing too fancy here, I don't get why this happens.

Can you tell me what could possibly be wrong ?

Thx

2

%{Subject->} is hitting issue 71.

3

Thx for your reply.

But, why is it an issue ? There is 9 members on the log and 9 field on my dissect. I don't get it :sweat_smile:

Edit: Ok I read the issue 71, so the padding without padding is the issue... But sometimes there is padding I want to remove.. neh, need to add a grok filter for that !
Thx Badger :wink:

4 
mutate { gsub => [ "message", "\s+", " " ] }

might be useful to strip out all the extra space.

2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.