If filebeat is deployed with single instance then no issue is observed in multiline handling. If Filebeat is deployed with daemonset then all events does not follow proper multiline pattern thus results in individual events. Is there any bug reported?

[Chitra1 (Chitra)]

If filebeat is deployed with single instance then no issue is observed in multiline handling. If Filebeat is deployed with daemonset then all events does not follow proper multiline pattern thus results in individual events. Is there any bug reported ?

[MichelLaterman]

Issues and bug reports are handled through the beats github repo: Issues · elastic/beats · GitHub

Can you provide any details on how filebeat is deployed and configured?

[Chitra1 (Chitra)]

Filebeat config looks like this

filebeat-container.yml: |2
filebeat.inputs:

• type: log
  paths: [/opt/abc/myfile.log]
  multiline.pattern: ^[1]
  multiline.negate: true
  multiline.match: after
  output.kafka:
  hosts: ${KAFKA_HOST}
  topic: myfilelogs
  compression: snappy
  max_message_bytes: 1000000
  logging.level: debug

Log file -
[00:00:37]
hsdhdjn : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.249/0.412/2.69
[00:00:44]
240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.249/0.316/0.390
[00:00:51]
240************** : xmt/rcv/%loss = 24/24/0%, min/avg/max = 0.260/0.3**/0.5**
[00:00:58]
240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.264/0.322/0.412
[00:01:05]
240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.242/0.372/0.859

output on kafka when using deployment -

{
"@timestamp": "2022-08-30T06:15:51.300Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.3"
},
"log": {
"offset": 518535,
"file": {
"path": "/home/obf/abc/myfile.log"
},
"flags": [
"multiline"
]
},
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "abc-network-tool-network***-0"
},
"agent": {
"version": "8.2.3",
"ephemeral_id": "",
"id": "2dc778780",
"name": "-0",
"type": "filebeat"
},
"message": "[00:01:05]\n240***** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.242/0.372/0.859"
}
output on kafka when using daemonset -

{
"@timestamp": "2022-08-30T06:15:51.300Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.3"
},
"log": {
"offset": 518535,
"file": {
"path": "/home/obf/abc/myfile.log"
}
},
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "abc-network-tool-network***-0"
},
"agent": {
"version": "8.2.3",
"ephemeral_id": "",
"id": "2dc778780",
"name": "-0",
"type": "filebeat"
},
"message": "[00:00:58]"
}

{
"@timestamp": "2022-08-30T06:15:51.300Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.3"
},
"log": {
"offset": 518536,
"file": {
"path": "/home/obf/abc/myfile.log"
}
},
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "abc-network-tool-network***-0"
},
"agent": {
"version": "8.2.3",
"ephemeral_id": "",
"id": "2dc778780",
"name": "-0",
"type": "filebeat"
},
"message": "240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.264/0.322/0.412"
}

---

1. \d{2}:\d{2}:\d{2} ↩︎