# Filebeat does not correctly merge multiline events **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 31, 2020, 5:04pm UTC](https://discuss.elastic.co/t/filebeat-does-not-correctly-merge-multiline-events/253907 "2020-10-31T17:04:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Sebastian\_Kenter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_kenter/32/78218_2.png) [@Sebastian\_Kenter](https://discuss.elastic.co/u/Sebastian_Kenter)\ **Post date:** [October 31, 2020, 5:04pm UTC](https://discuss.elastic.co/t/filebeat-does-not-correctly-merge-multiline-events/253907/1 "2020-10-31T17:04:44Z")
I have the following problem: This is my logfile ``` 11:30:00,909 9=DEB BswTcpTskComSrv::passTele() 11:30:00,909 12=EVT | TELEGRAMM SEND comsrv->TskEvm: |TskBrm|TskEvm|TeleDb| 11:30:00,909 12=EVT | BswTcpTskComSrv::sendTeleTo:23 Bytes gesendet 11:30:00,909 11:30:00,909 LOAD STATISTICS 11:30:00,909 11:30:00,909 =========================+================================+============+========================================+===================== 11:30:00,909 | % BUSY | TOTAL | TELEGRAMS | 11:30:00,909 TASK | MIN 10MIN HOUR TOTAL | TIME [s] | MIN 10MIN HOUR TOTAL OPEN | UP SINCE 11:30:00,909 -------------------------+--------------------------------+------------+----------------------------------------+--------------------- 11:30:00,909 DEEFR0004 | 0.20 0.18 0.18 0.23 | 0.433 | 2 6 6 6 0 | 29.10.2020 11:26:52 11:30:00,909 DEEFR0046 | 0.00 0.00 0.02 0.28 | 44.440 | 0 0 10 550 0 | 29.10.2020 07:01:54 11:30:00,909 DEEFR0051 | 0.76 0.98 1.21 1.27 | 54.578 | 2 64 294 436 0 | 29.10.2020 10:18:37 11:30:00,910 DEEFR0082 | 0.00 3.70 2.80 1.68 | 356.703 | 0 294 1304 3988 0 | 29.10.2020 05:36:39 11:30:00,910 DEEFR0083 | 0.00 0.00 0.00 0.10 | 20.151 | 0 0 0 280 0 | 29.10.2020 05:37:32 11:30:00,910 DEEFR0086 | 0.00 0.00 0.00 0.21 | 33.894 | 0 0 0 350 0 | 29.10.2020 06:59:41 11:30:00,910 DEEFR0093 | 0.00 0.00 0.00 0.24 | 39.455 | 0 0 0 480 0 | 29.10.2020 07:00:33 11:30:00,910 DEEFR0102 | 0.00 0.00 0.00 0.45 | 94.647 | 0 0 0 1166 0 | 29.10.2020 05:36:30 11:30:00,910 DEEFR0107 | 0.00 0.22 0.15 0.23 | 39.704 | 0 16 84 506 0 | 29.10.2020 06:40:16 11:30:00,910 DEEFR0114 | 0.00 0.09 0.66 0.64 | 73.202 | 0 8 288 854 0 | 29.10.2020 08:18:58 11:30:00,910 DEEFR0130 | 0.00 0.00 0.00 0.07 | 11.304 | 0 0 0 180 0 | 29.10.2020 07:04:53 11:30:00,910 DEEFR0141 | 0.00 0.00 0.00 0.00 | 0.292 | 0 0 0 2 0 | 29.10.2020 08:30:10 11:30:00,910 DEEFR0147 | 0.00 1.13 0.71 0.91 | 55.629 | 0 90 294 646 0 | 29.10.2020 09:48:26 11:30:00,910 DEEFR0164 | 0.00 0.00 0.00 0.13 | 15.560 | 0 0 0 182 0 | 29.10.2020 08:03:41 11:30:00,910 DEEFR0165 | 0.00 0.64 0.17 0.11 | 23.701 | 0 18 40 314 0 | 29.10.2020 05:35:51 11:30:00,910 DEEFR0168 | 0.70 0.12 0.69 0.63 | 46.330 | 6 12 230 498 0 | 29.10.2020 09:28:01 11:30:00,910 DEEFR0169 | 0.84 1.21 1.21 1.28 | 7.253 | 8 88 88 88 0 | 29.10.2020 11:20:34 11:30:00,910 DEEFRK023 | 0.92 2.48 2.48 2.64 | 11.889 | 4 146 146 146 0 | 29.10.2020 11:22:30 11:30:00,910 DEEFRK035 | 0.00 0.08 0.08 0.13 | 0.102 | 0 2 2 2 0 | 29.10.2020 11:28:40 11:30:00,910 DEEFRK062 | 0.00 0.00 0.00 0.36 | 37.763 | 0 0 0 286 0 | 29.10.2020 08:36:27 11:30:00,910 PSI-MDT4.0 | 0.00 0.00 0.00 0.00 | 0.000 | 0 0 0 0 0 | 22.10.2020 11:10:30 11:30:00,912 -------------------------+--------------------------------+------------+----------------------------------------+--------------------- 11:30:00,912 ALL | 100.00 100.00 100.00 100.00 | 605963.931 | 0 0 0 0 0 | 22.10.2020 11:10:19 11:30:00,912 =========================+================================+============+========================================+===================== 11:30:00,912 11:30:00,912 34 CLIENTS CONNECTED. 11:30:00,912 11:30:00,912 8=EVT TELEGRAMM RECV TskBrm->comsrv: |TskBrm|TskSls|TeleDb| 11:30:00,912 9=DEB BswTcpTskComSrv::passTele() 11:30:00,912 12=EVT | TELEGRAMM SEND comsrv->TskSls: |TskBrm|TskSls|TeleDb| 11:30:00,912 12=EVT | BswTcpTskComSrv::sendTeleTo:23 Bytes gesendet ``` Every line in that logfile is an event to sent to logstash and then transformed in some way. This works fine. But what I want to archive is that the block between **11:30:00,909 LOAD STATISTICS** and **11:30:00,912 34 CLIENTS CONNECTED.** should be send as one message (as a side note: this statistic repeats every minute). This is my filebeat config: ``` - type: log enabled: true paths: - /fpsi_logs/*.comsrv.prot.* fields: fpsi_task: 'comsrv' multiline.type: pattern multiline.pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s\b[A-Z]{4}\b\s\b[A-Z]{10}' multiline.negate: true multiline.match: after multiline.flush_pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s[0-9]{1,3}\s[A-Z]{7}\s[A-Z]{9}' ``` What happens is, that all "Load Statistics" get send as separate messages but everything between two of them ist aggregated as one message as well. [Output from Kibana](https://i.stack.imgur.com/oUHCa.png) At the moment I don't care of \_grokparsefailures, just looking for a config to archive the result I need. Someone here to help me? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [November 3, 2020, 6:14pm UTC](https://discuss.elastic.co/t/filebeat-does-not-correctly-merge-multiline-events/253907/2 "2020-11-03T18:14:15Z")
Hey @Sebastian_Kenter, So you want to drop the events without the load statistics? Something you might try is to use a processor to drop the events that you don't want. For example something like this: ```auto processors: - drop_event: when: not.contains.message: "LOAD STATISTICS" ``` ---
**Author:** ![Sebastian\_Kenter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_kenter/32/78218_2.png) [@Sebastian\_Kenter](https://discuss.elastic.co/u/Sebastian_Kenter)\ **Post date:** [November 3, 2020, 9:14pm UTC](https://discuss.elastic.co/t/filebeat-does-not-correctly-merge-multiline-events/253907/3 "2020-11-03T21:14:33Z")
Hi @jsoriano, no the other way round. I want to keep everything as separate messages except the "Load Statistics" which should be merged as single event. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [November 4, 2020, 11:09am UTC](https://discuss.elastic.co/t/filebeat-does-not-correctly-merge-multiline-events/253907/4 "2020-11-04T11:09:44Z")
Oh, I see It seems that all the other events start with something like `11:30:00,912 8=EVT...`, maybe you can modify the pattern to match both kind of lines: - The ones starting with a time stamp plus `LOAD STATISTICS` - The ones starting with a time stamp plus an event (that seem to match a regexp like `[\s0-9]=[A-Z]{3}`). Then maybe you don't need the flush pattern. It would be something like this (not tested): ```auto multiline.pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s\b(LOAD STATISTICS|[\s0-9]+=[A-Z]{3})' multiline.negate: true multiline.match: after ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 2, 2020, 1:09pm UTC](https://discuss.elastic.co/t/filebeat-does-not-correctly-merge-multiline-events/253907/5 "2020-12-02T13:09:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.