Hi Everyone,
I couldn't find a reason, why Filebeat is going into the loop with "Unable to find match for dissect pattern" when it reaches the end of the file with filestream input mode. I'm parsing multiple very similar files, created by the same software, just to different files, with different flags. Most of them are parsed properly, but some of them are causing output like this. More interesting is, that I see in the filebeat debug output, it went through all lines and correctly reached the end of the file.
After that it is producing a lot of the same logs I've pasted below.
I will appreciate any help, thanks!
"message":"Bulk item insert failed (i=11, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}"
This is my filebeat config:
filebeat.inputs:
- type: filestream
id: test-logs
enabled: true
paths:
- /home/test/writer/log/current
output.elasticsearch:
hosts: ["http://dc-prd-mon02:9200"]
indices:
- index: "test-logs"
when.contains:
log.file.path: "test"
pipelines:
- pipeline: "test-logs-pipeline"
when.contains:
log.file.path: "test"
Ingest pipeline:
PUT _ingest/pipeline/test-logs-pipeline
{
"processors": [
{
"dissect": {
"field": "message",
"pattern": "%{date} %{+date} %{message}"
}
},
{
"date": {
"field": "date",
"formats": [
"yyyy-MM-dd HH:mm:ss,SSS",
"yyyy-MM-ddHH:mm:ss,SSS"
]
}
},
{
"remove": {
"field": "date"
}
}
],
"on_failure": [
{
"dissect": {
"field": "message",
"pattern": "%{message}",
"ignore_failure": true
}
}
]
}
Output from the filebeat debug
{"log.level":"debug","@timestamp":"2023-05-19T04:31:41.812-0500","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /home/test/writer/log/current; Backoff now.","service.name":"filebeat","id":"test-logs","source_file":"filestream::test-logs::native::70328469-65028","path":"/home/test/writer/log/current","state-id":"native::70328469-65028","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.765-0500","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(http://dc-prd-mon02:9200))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.765-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":267},"message":"ES Ping(url=http://dc-prd-mon02:9200)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.766-0500","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":42},"message":"Completed dialing successfully","service.name":"filebeat","network":"tcp","address":"dc-prd-mon02:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.768-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"Ping status code: 200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.768-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.768-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET http://dc-prd-mon02:9200/_license?human=false <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.769-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":267},"message":"ES Ping(url=http://dc-prd-mon02:9200)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.770-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"Ping status code: 200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.770-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.770-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET http://dc-prd-mon02:9200/ <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.771-0500","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.771-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET http://dc-prd-mon02:9200/_ilm/policy/filebeat <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.778-0500","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.778-0500","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.779-0500","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"HEAD http://dc-prd-mon02:9200/_index_template/filebeat <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.814-0500","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"filebeat\" already exists and will not be overwritten.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.814-0500","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-19T04:31:42.814-0500","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://dc-prd-mon02:9200)) established","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":247},"message":"PublishEvents: 225 events have been published to elasticsearch in 72.974084ms.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=6, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=11, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=13, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=15, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=17, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=19, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:42.887-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":435},"message":"Bulk item insert failed (i=21, status=500): {\"type\":\"find_match\",\"reason\":\"find_match: Unable to find match for dissect pattern: %{message} against source: \"}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-05-19T04:31:43.812-0500","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /home/test/writer/log/current; Backoff now.","service.name":"filebeat","id":"test-logs","source_file":"filestream::test-logs::native::70328469-65028","path":"/home/test/writer/log/current","state-id":"native::70328469-65028","ecs.version":"1.6.0"}