Filebeat unable to find match for dissect pattern

obol89 · May 19, 2023, 9:49am

Hi Everyone,

I couldn't find a reason, why Filebeat is going into the loop with "Unable to find match for dissect pattern" when it reaches the end of the file with filestream input mode. I'm parsing multiple very similar files, created by the same software, just to different files, with different flags. Most of them are parsed properly, but some of them are causing output like this. More interesting is, that I see in the filebeat debug output, it went through all lines and correctly reached the end of the file.
After that it is producing a lot of the same logs I've pasted below.
I will appreciate any help, thanks!

"message":"Bulk item insert failed (i=11, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}"

This is my filebeat config:

filebeat.inputs:

- type: filestream
  id: test-logs
  enabled: true
  paths:
    - /home/test/writer/log/current
    
output.elasticsearch:
  hosts: ["http://dc-prd-mon02:9200"]
  indices:
  - index: "test-logs"
    when.contains:
      log.file.path: "test"
  pipelines:
  - pipeline: "test-logs-pipeline"
    when.contains:
      log.file.path: "test"

Ingest pipeline:

PUT _ingest/pipeline/test-logs-pipeline
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{date} %{+date} %{message}"
      }
    },
    {
      "date": {
        "field": "date",
        "formats": [
          "yyyy-MM-dd HH:mm:ss,SSS",
          "yyyy-MM-ddHH:mm:ss,SSS"
        ]
      }
    },
    {
      "remove": {
        "field": "date"
      }
    }
  ],
  "on_failure": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{message}",
        "ignore_failure": true
      }
    }
  ]
}

Output from the filebeat debug

{"log.level":"debug","@timestamp":"2023-05-19T04:31:41.812-0500","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /home/test/writer/log/current; Backoff now.","service.name":"filebeat","id":"test-logs","source_file":"filestream::test-logs::native::70328469-65028","path":"/home/test/writer/log/current","state-id":"native::70328469-65028","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.765-0500","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(http://dc-prd-mon02:9200))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.765-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":267},"message":"ES Ping(url=http://dc-prd-mon02:9200)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.766-0500","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":42},"message":"Completed dialing successfully","service.name":"filebeat","network":"tcp","address":"dc-prd-mon02:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.768-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"Ping status code: 200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.768-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.768-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET http://dc-prd-mon02:9200/_license?human=false  <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.769-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":267},"message":"ES Ping(url=http://dc-prd-mon02:9200)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.770-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"Ping status code: 200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.770-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.770-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET http://dc-prd-mon02:9200/  <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.771-0500","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.771-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET http://dc-prd-mon02:9200/_ilm/policy/filebeat  <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.778-0500","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.778-0500","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.779-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"HEAD http://dc-prd-mon02:9200/_index_template/filebeat  <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.814-0500","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"filebeat\" already exists and will not be overwritten.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.814-0500","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.814-0500","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://dc-prd-mon02:9200)) established","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":247},"message":"PublishEvents: 225 events have been published to elasticsearch in 72.974084ms.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=6, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=11, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=13, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=15, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=17, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=19, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=21, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:43.812-0500","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /home/test/writer/log/current; Backoff now.","service.name":"filebeat","id":"test-logs","source_file":"filestream::test-logs::native::70328469-65028","path":"/home/test/writer/log/current","state-id":"native::70328469-65028","ecs.version":"1.6.0"}

obol89 · May 30, 2023, 1:30pm

Does anyone know how the "source" can be empty?

against source: \"

leandrojmp · May 30, 2023, 1:38pm

An empty line or a line with just a space in your source file could result on this.

The issue is in your source file.

system · June 27, 2023, 3:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
About FileBeat Dissect processor Beats filebeat	2	870	October 28, 2022
Dissect failure for no obvious reason Logstash	4	1193	October 29, 2020
Dissect combined with multiline pattern gives errors Beats filebeat	6	3068	August 5, 2020
Error No matching indices found: No indices match pattern "filebeat-*" Beats filebeat	5	4967	July 2, 2018
Eror : No indices match pattern “filebeat-*” - filebeat 8.1.2 Beats filebeat	1	273	May 9, 2022

Filebeat unable to find match for dissect pattern

Related topics