If filebeat is deployed with single instance then no issue is observed in multiline handling. If Filebeat is deployed with daemonset then all events does not follow proper multiline pattern thus results in individual events. Is there any bug reported ?
Issues and bug reports are handled through the beats github repo: Issues · elastic/beats · GitHub
Can you provide any details on how filebeat is deployed and configured?
Filebeat config looks like this
filebeat-container.yml: |2
filebeat.inputs:
- type: log
paths: [/opt/abc/myfile.log]
multiline.pattern: [1]
multiline.negate: true
multiline.match: after
output.kafka:
hosts: ${KAFKA_HOST}
topic: myfilelogs
compression: snappy
max_message_bytes: 1000000
logging.level: debug
Log file -
[00:00:37]
hsdhdjn : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.249/0.412/2.69
[00:00:44]
240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.249/0.316/0.390
[00:00:51]
240************** : xmt/rcv/%loss = 24/24/0%, min/avg/max = 0.260/0.3**/0.5**
[00:00:58]
240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.264/0.322/0.412
[00:01:05]
240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.242/0.372/0.859
output on kafka when using deployment -
{
"@timestamp": "2022-08-30T06:15:51.300Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.3"
},
"log": {
"offset": 518535,
"file": {
"path": "/home/obf/abc/myfile.log"
},
"flags": [
"multiline"
]
},
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "abc-network-tool-network***-0"
},
"agent": {
"version": "8.2.3",
"ephemeral_id": "",
"id": "2dc778780",
"name": "-0",
"type": "filebeat"
},
"message": "[00:01:05]\n240***** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.242/0.372/0.859"
}
output on kafka when using daemonset -
{
"@timestamp": "2022-08-30T06:15:51.300Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.3"
},
"log": {
"offset": 518535,
"file": {
"path": "/home/obf/abc/myfile.log"
}
},
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "abc-network-tool-network***-0"
},
"agent": {
"version": "8.2.3",
"ephemeral_id": "",
"id": "2dc778780",
"name": "-0",
"type": "filebeat"
},
"message": "[00:00:58]"
}
{
"@timestamp": "2022-08-30T06:15:51.300Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.3"
},
"log": {
"offset": 518536,
"file": {
"path": "/home/obf/abc/myfile.log"
}
},
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "abc-network-tool-network***-0"
},
"agent": {
"version": "8.2.3",
"ephemeral_id": "",
"id": "2dc778780",
"name": "-0",
"type": "filebeat"
},
"message": "240************** : xmt/rcv/%loss = 23/23/0%, min/avg/max = 0.264/0.322/0.412"
}
-
\d{2}:\d{2}:\d{2} ↩︎
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.