Dissect filter sometimes not working

Elastic Stack Logstash
1

Logstash cannot find the pattern in my log. It sometimes can dissect successfully but sometimes not.

[2021-07-29T04:18:35,253][WARN ][org.logstash.dissect.Dissector][main][0b28d6955374719d4eec09ce396e5505f458a3d25e49b4066f5ec97f5bd76281] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{ts} %{log_level} %{component_name} %{guid}: %{msg}", "event"=>{"log"=>{"file"=>{"path"=>"/usr/share/dockerlogs/data/27be5e813cf8971fba1c9d6150178284cd72951851c1d316fb146d6ac793c988/27be5e813cf8971fba1c9d6150178284cd72951851c1d316fb146d6ac793c988-json.log"}, "offset"=>1623442}, "agent"=>{"ephemeral_id"=>"a3facc33-24f8-4f35-a666-cbb9e300c215", "name"=>"filebeat", "id"=>"82a9cd8d-1df1-4839-b1e0-16386fb81e35", "version"=>"7.10.2", "hostname"=>"filebeat", "type"=>"filebeat"}, "message"=>"2021-07-29T04:18:32.656Z DEBUG API_Gateway 863dd9dc-da1e-4dbd-8c8a-45f18b879c75 GET /report/generated from IP ::ffff:172.17.0.1 ", "input"=>{"type"=>"docker"}, "@timestamp"=>2021-07-29T04:18:32.657Z, "@version"=>"1", "ecs"=>{"version"=>"1.6.0"}, "stream"=>"stdout", "container"=>{"name"=>"gateway-dev", "image"=>{"name"=>"maindev.azurecr.io/api-gateway-dev:build_26"}, "id"=>"27be5e813cf8971fba1c9d6150178284cd72951851c1d316fb146d6ac793c988"}, "host"=>{"name"=>"filebeat"}, "tags"=>["beats_input_codec_plain_applied", "_dissectfailure"]}}

Here is what my log looks like:

2021-07-29T04:18:32.656Z DEBUG API_Gateway 863dd9dc-da1e-4dbd-8c8a-45f18b879c75 GET /report/generated from IP ::ffff:172.17.0.1

Here is my logstash.conf:

input { 
	beats {
	    port => 5044
  	}
} 
filter {
    if [container][name] =~ "docker-elk"  or [container][name] =~ "docker-jenkins" {
        drop { }
    }
    dissect {
        mapping => { "message" => "%{?ts} %{log_level} %{component_name} %{guid}: %{?msg}" }
    }
}
output { 
	elasticsearch { 
		hosts => "elasticsearch:9200"
		manage_template => false
    	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
	} 
	
    stdout { codec => rubydebug }
}

Thanks in advance.

2

There is no colon after the guid. And there is no space after the colon

 863dd9dc-da1e-4dbd-8c8a-45f18b879c75 GET /report/generated from IP ::ffff

so

%{guid}: %{msg}"

cannot match. Removing the colon from the mapping should fix it.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.