Dissector mapping, pattern not found

Elastic Stack Logstash
1

Can't understand why my pattern doesn't work. None of events are parsed with such error:

[2019-06-19T18:01:29,601][WARN ][org.logstash.dissect.Dissector] Dissector mapping, pattern not found 
{
"field"=>"message", 
"pattern"=>"%{dateTime} %{+dateTime} %{} Host:\\\"%{host}\\\" Zone:\\\"%{zone}\\\" \\\"%{adapter}\\\" \\\"%{process}\\\" %{process_id} %{header}\\n%{?content}",
"event"=>{"tags"=>["beats_input_codec_plain_applied", "_dissectfailure"], 
"message"=>"19.06.2019 00:01:35.611 101 Host:\"tst-***\" Zone:\"Test\" \"Adp_123\" \"Processes/Out/loyalty/setLoyalty.rq\" 65f88369-7f4a-44c9-be85-1d5905b0f80f Start proccess\n<document xsi:schemaLocation=\"\"...", 
"ecs"=>{"version"=>"1.0.0"}, 
"@version"=>"1", 
"@timestamp"=>2019-06-19T15:01:28.125Z, 
"input"=>{"type"=>"log"}, 
"host"=>{"id"=>"f5931da5-b559-4a6f-b521-47236c027043", "os"=>{"platform"=>"windows","version"=>"6.1", "name"=>"***", 
"kernel"=>"6.1.7601.24441 (win7sp1_ldr.190418-1735)", "family"=>"windows", uild"=>"7601.24443"}, 
"name"=>"***", 
"architecture"=>"x86_64", 
"hostname"=>"*"}, 
"agent"=>{"id"=>"addda848-82c9-4fc1-a90c-490aa71687a9", 
"type"=>"filebeat", 
"ephemeral_id"=>"2536c3a9-ba85-4b14-9c29-2b010adb917b", 
"version"=>"7.1.1", 
"hostname"=>"*"}, 
"log"=>{"offset"=>881, "file"=>{"path"=>"***"},
"flags"=>["multiline"]}}
}

Logstash filter:

input {
    beats {
        port => "5044"
    }
}

filter {
    dissect {
      mapping => {
        "message" => "%{dateTime} %{+dateTime} %{} Host:\"%{host}\" Zone:\"%{zone}\" \"%{adapter}\" \"%{process}\" %{process_id} %{header}\n%{?content}"
      }
    }
}

Input is a filebeat event with multiline message. Eg:

  Publish event: {
  "@timestamp": "2019-06-19T15:13:59.844Z",
  "@metadata": {
    "beat": "",
    "type": "_doc",
    "version": ""
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "***",
    "architecture": "x86_64",
    "os": {
      "platform": "windows",
      "version": "6.1",
      "family": "windows",
      "name": "***",
      "kernel": "6.1.7601.24441 (win7sp1_ldr.190418-1735)",
      "build": "7601.24443"
    },
    "id": "f5931da5-b559-4a6f-b521-47236c027043",
    "hostname": "***"
  },
  "agent": {
    "ephemeral_id": "6d7e9c10-b4de-401d-afc2-b3a4e131a392",
    "hostname": "***",
    "id": "addda848-82c9-4fc1-a90c-490aa71687a9",
    "version": "7.1.1",
    "type": "filebeat"
  },
  "log": {
    "flags": [
      "multiline"
    ],
    "offset": 1266,
    "file": {
      "path": "***"
    }
  },
  "message": "18.06.2019 13:30:20.084 2 Host:\"tst-***\" Zone:\"Test123\" \"Adp_EVAM_In_root\" \"Processes/In/Agreement/Set.Loyalty\" 6e0de059-3166-48ef-bfda-ad1ad6229c2b Sending request\n
<setLoyaltyRequest>\n    <clientID>12345</clientID>\n    <loyaltyCode>string</loyaltyCode>\n</setLoyaltyRequest>\n\n",
  "input": {
    "type": "log"
  }
}

My best guess that it is something with char escaping or newline symbol

2

Correct! Try

    dissect { mapping => { "message" => '%{dateTime} %{+dateTime} %{} Host:"%{host}" Zone:"%{zone}" "%{adapter}" "%{process}" %{process_id} %{header}
%{?content}' } }

That is, use single quotes around the pattern so that you do not have to escape the double quotes, and use a literal newline in the pattern string.

1 Like
3

Thanks! That helped. Also found this config that is false by default.

# When enabled, process escaped characters such as \n and \" in strings in the
# pipeline configuration files.
#
 config.support_escapes: true
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.