Filebeat does not correctly merge multiline events

I have the following problem:

This is my logfile

11:30:00,909  9=DEB BswTcpTskComSrv::passTele()
11:30:00,909 12=EVT |  TELEGRAMM SEND comsrv->TskEvm: |TskBrm|TskEvm|TeleDb|
11:30:00,909 12=EVT |  BswTcpTskComSrv::sendTeleTo:23 Bytes gesendet
11:30:00,909 
11:30:00,909 LOAD STATISTICS
11:30:00,909 
11:30:00,909 =========================+================================+============+========================================+=====================
11:30:00,909                          |             % BUSY             |   TOTAL    |                TELEGRAMS               |
11:30:00,909 TASK                     |    MIN   10MIN    HOUR   TOTAL |  TIME [s]  |    MIN   10MIN    HOUR   TOTAL    OPEN | UP SINCE
11:30:00,909 -------------------------+--------------------------------+------------+----------------------------------------+---------------------
11:30:00,909 DEEFR0004                |   0.20    0.18    0.18    0.23 |      0.433 |      2       6       6       6       0 | 29.10.2020 11:26:52
11:30:00,909 DEEFR0046                |   0.00    0.00    0.02    0.28 |     44.440 |      0       0      10     550       0 | 29.10.2020 07:01:54
11:30:00,909 DEEFR0051                |   0.76    0.98    1.21    1.27 |     54.578 |      2      64     294     436       0 | 29.10.2020 10:18:37
11:30:00,910 DEEFR0082                |   0.00    3.70    2.80    1.68 |    356.703 |      0     294    1304    3988       0 | 29.10.2020 05:36:39
11:30:00,910 DEEFR0083                |   0.00    0.00    0.00    0.10 |     20.151 |      0       0       0     280       0 | 29.10.2020 05:37:32
11:30:00,910 DEEFR0086                |   0.00    0.00    0.00    0.21 |     33.894 |      0       0       0     350       0 | 29.10.2020 06:59:41
11:30:00,910 DEEFR0093                |   0.00    0.00    0.00    0.24 |     39.455 |      0       0       0     480       0 | 29.10.2020 07:00:33
11:30:00,910 DEEFR0102                |   0.00    0.00    0.00    0.45 |     94.647 |      0       0       0    1166       0 | 29.10.2020 05:36:30
11:30:00,910 DEEFR0107                |   0.00    0.22    0.15    0.23 |     39.704 |      0      16      84     506       0 | 29.10.2020 06:40:16
11:30:00,910 DEEFR0114                |   0.00    0.09    0.66    0.64 |     73.202 |      0       8     288     854       0 | 29.10.2020 08:18:58
11:30:00,910 DEEFR0130                |   0.00    0.00    0.00    0.07 |     11.304 |      0       0       0     180       0 | 29.10.2020 07:04:53
11:30:00,910 DEEFR0141                |   0.00    0.00    0.00    0.00 |      0.292 |      0       0       0       2       0 | 29.10.2020 08:30:10
11:30:00,910 DEEFR0147                |   0.00    1.13    0.71    0.91 |     55.629 |      0      90     294     646       0 | 29.10.2020 09:48:26
11:30:00,910 DEEFR0164                |   0.00    0.00    0.00    0.13 |     15.560 |      0       0       0     182       0 | 29.10.2020 08:03:41
11:30:00,910 DEEFR0165                |   0.00    0.64    0.17    0.11 |     23.701 |      0      18      40     314       0 | 29.10.2020 05:35:51
11:30:00,910 DEEFR0168                |   0.70    0.12    0.69    0.63 |     46.330 |      6      12     230     498       0 | 29.10.2020 09:28:01
11:30:00,910 DEEFR0169                |   0.84    1.21    1.21    1.28 |      7.253 |      8      88      88      88       0 | 29.10.2020 11:20:34
11:30:00,910 DEEFRK023                |   0.92    2.48    2.48    2.64 |     11.889 |      4     146     146     146       0 | 29.10.2020 11:22:30
11:30:00,910 DEEFRK035                |   0.00    0.08    0.08    0.13 |      0.102 |      0       2       2       2       0 | 29.10.2020 11:28:40
11:30:00,910 DEEFRK062                |   0.00    0.00    0.00    0.36 |     37.763 |      0       0       0     286       0 | 29.10.2020 08:36:27
11:30:00,910 PSI-MDT4.0               |   0.00    0.00    0.00    0.00 |      0.000 |      0       0       0       0       0 | 22.10.2020 11:10:30
11:30:00,912 -------------------------+--------------------------------+------------+----------------------------------------+---------------------
11:30:00,912 ALL                      | 100.00  100.00  100.00  100.00 | 605963.931 |      0       0       0       0       0 | 22.10.2020 11:10:19
11:30:00,912 =========================+================================+============+========================================+=====================
11:30:00,912 
11:30:00,912 34 CLIENTS CONNECTED. 
11:30:00,912 
11:30:00,912  8=EVT TELEGRAMM RECV TskBrm->comsrv: |TskBrm|TskSls|TeleDb|
11:30:00,912  9=DEB BswTcpTskComSrv::passTele()
11:30:00,912 12=EVT |  TELEGRAMM SEND comsrv->TskSls: |TskBrm|TskSls|TeleDb|
11:30:00,912 12=EVT |  BswTcpTskComSrv::sendTeleTo:23 Bytes gesendet

Every line in that logfile is an event to sent to logstash and then transformed in some way. This works fine.

But what I want to archive is that the block between 11:30:00,909 LOAD STATISTICS and 11:30:00,912 34 CLIENTS CONNECTED. should be send as one message (as a side note: this statistic repeats every minute).

This is my filebeat config:

- type: log
  enabled: true
  paths:
    - /fpsi_logs/*.comsrv.prot.*
  fields:
    fpsi_task: 'comsrv'
  multiline.type: pattern
  multiline.pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s\b[A-Z]{4}\b\s\b[A-Z]{10}'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s[0-9]{1,3}\s[A-Z]{7}\s[A-Z]{9}'

What happens is, that all "Load Statistics" get send as separate messages but everything between two of them ist aggregated as one message as well.

Output from Kibana

At the moment I don't care of _grokparsefailures, just looking for a config to archive the result I need.

Someone here to help me?

Hey @Sebastian_Kenter,

So you want to drop the events without the load statistics?

Something you might try is to use a processor to drop the events that you don't want. For example something like this:

  processors:
    - drop_event:
        when:
          not.contains.message: "LOAD STATISTICS"

Hi @jsoriano,

no the other way round. I want to keep everything as separate messages except the "Load Statistics" which should be merged as single event.

Oh, I see

It seems that all the other events start with something like 11:30:00,912 8=EVT..., maybe you can modify the pattern to match both kind of lines:

  • The ones starting with a time stamp plus LOAD STATISTICS
  • The ones starting with a time stamp plus an event (that seem to match a regexp like [\s0-9]=[A-Z]{3}).

Then maybe you don't need the flush pattern. It would be something like this (not tested):

multiline.pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s\b(LOAD STATISTICS|[\s0-9]+=[A-Z]{3})'
multiline.negate: true
multiline.match: after

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.