Multiline parsing issue

Elastic Stack Beats
1

Hi all,
filebeat newbie here.
I'm trying to reduce this log:

[2020-04-05T00:20:00] /usr/bin/rsnapshot -c /etc/rsnapshot_nuc.conf alpha: started
[2020-04-05T00:20:00] echo 2123427 > /var/run/rsnapshot_nuc.pid
[2020-04-05T00:20:00] /bin/rm -rf /mnt/bck_nuc/rsnapshot/alpha.59/
[2020-04-05T00:20:13] mv /mnt/bck_nuc/rsnapshot/alpha.58/ /mnt/bck_nuc/rsnapshot/alpha.59/
[2020-04-05T00:20:13] mv /mnt/bck_nuc/rsnapshot/alpha.57/ /mnt/bck_nuc/rsnapshot/alpha.58/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.56/ /mnt/bck_nuc/rsnapshot/alpha.57/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.55/ /mnt/bck_nuc/rsnapshot/alpha.56/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.54/ /mnt/bck_nuc/rsnapshot/alpha.55/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.53/ /mnt/bck_nuc/rsnapshot/alpha.54/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.52/ /mnt/bck_nuc/rsnapshot/alpha.53/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.51/ /mnt/bck_nuc/rsnapshot/alpha.52/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.50/ /mnt/bck_nuc/rsnapshot/alpha.51/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.49/ /mnt/bck_nuc/rsnapshot/alpha.50/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.48/ /mnt/bck_nuc/rsnapshot/alpha.49/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.47/ /mnt/bck_nuc/rsnapshot/alpha.48/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.46/ /mnt/bck_nuc/rsnapshot/alpha.47/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.45/ /mnt/bck_nuc/rsnapshot/alpha.46/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.44/ /mnt/bck_nuc/rsnapshot/alpha.45/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.43/ /mnt/bck_nuc/rsnapshot/alpha.44/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.42/ /mnt/bck_nuc/rsnapshot/alpha.43/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.41/ /mnt/bck_nuc/rsnapshot/alpha.42/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.40/ /mnt/bck_nuc/rsnapshot/alpha.41/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.39/ /mnt/bck_nuc/rsnapshot/alpha.40/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.38/ /mnt/bck_nuc/rsnapshot/alpha.39/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.37/ /mnt/bck_nuc/rsnapshot/alpha.38/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.36/ /mnt/bck_nuc/rsnapshot/alpha.37/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.35/ /mnt/bck_nuc/rsnapshot/alpha.36/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.34/ /mnt/bck_nuc/rsnapshot/alpha.35/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.33/ /mnt/bck_nuc/rsnapshot/alpha.34/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.32/ /mnt/bck_nuc/rsnapshot/alpha.33/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.31/ /mnt/bck_nuc/rsnapshot/alpha.32/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.30/ /mnt/bck_nuc/rsnapshot/alpha.31/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.29/ /mnt/bck_nuc/rsnapshot/alpha.30/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.28/ /mnt/bck_nuc/rsnapshot/alpha.29/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.27/ /mnt/bck_nuc/rsnapshot/alpha.28/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.26/ /mnt/bck_nuc/rsnapshot/alpha.27/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.25/ /mnt/bck_nuc/rsnapshot/alpha.26/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.24/ /mnt/bck_nuc/rsnapshot/alpha.25/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.23/ /mnt/bck_nuc/rsnapshot/alpha.24/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.22/ /mnt/bck_nuc/rsnapshot/alpha.23/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.21/ /mnt/bck_nuc/rsnapshot/alpha.22/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.20/ /mnt/bck_nuc/rsnapshot/alpha.21/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.19/ /mnt/bck_nuc/rsnapshot/alpha.20/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.18/ /mnt/bck_nuc/rsnapshot/alpha.19/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.17/ /mnt/bck_nuc/rsnapshot/alpha.18/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.16/ /mnt/bck_nuc/rsnapshot/alpha.17/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.15/ /mnt/bck_nuc/rsnapshot/alpha.16/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.14/ /mnt/bck_nuc/rsnapshot/alpha.15/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.13/ /mnt/bck_nuc/rsnapshot/alpha.14/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.12/ /mnt/bck_nuc/rsnapshot/alpha.13/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.11/ /mnt/bck_nuc/rsnapshot/alpha.12/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.10/ /mnt/bck_nuc/rsnapshot/alpha.11/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.9/ /mnt/bck_nuc/rsnapshot/alpha.10/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.8/ /mnt/bck_nuc/rsnapshot/alpha.9/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.7/ /mnt/bck_nuc/rsnapshot/alpha.8/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.6/ /mnt/bck_nuc/rsnapshot/alpha.7/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.5/ /mnt/bck_nuc/rsnapshot/alpha.6/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.4/ /mnt/bck_nuc/rsnapshot/alpha.5/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.3/ /mnt/bck_nuc/rsnapshot/alpha.4/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.2/ /mnt/bck_nuc/rsnapshot/alpha.3/
[2020-04-05T00:20:14] mv /mnt/bck_nuc/rsnapshot/alpha.1/ /mnt/bck_nuc/rsnapshot/alpha.2/
[2020-04-05T00:20:14] /bin/cp -al /mnt/bck_nuc/rsnapshot/alpha.0 /mnt/bck_nuc/rsnapshot/alpha.1
[2020-04-05T00:20:17] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/etc/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:17] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/root/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:17] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/home/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:18] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/var/unbound/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:18] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/office_fs/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:19] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/mnt/500/bck_bth/alpha.0/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:21] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/mnt/500/bck_daily/alpha.0/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:25] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/mnt/500/bck_ts10/alpha.0/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:31] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/mnt/500/bck_w10server/alpha.0/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:31] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --rsh=/usr/bin/ssh root@192.168.100.1:/mnt/500/bck_weekly/alpha.0/ /mnt/bck_nuc/rsnapshot/alpha.0/nuc/
[2020-04-05T00:20:33] touch /mnt/bck_nuc/rsnapshot/alpha.0/
[2020-04-05T00:20:33] rm -f /var/run/rsnapshot_nuc.pid
[2020-04-05T00:20:33] /usr/bin/rsnapshot -c /etc/rsnapshot_nuc.conf alpha: completed successfully

obtaining just the first and the last line, like this:

[2020-04-05T00:20:00] /usr/bin/rsnapshot -c /etc/rsnapshot_nuc.conf alpha: started
[2020-04-05T00:20:33] /usr/bin/rsnapshot -c /etc/rsnapshot_nuc.conf alpha: completed successfully

Filebeat (v7.6.2) processing files with output to graylog.
I already tried every config, multiline examples and eventually The Go Playgroud, without success.
Relevant config from /etc/filebeat/filebeat.yml file:

fields_under_root: true
fields.source: v2
...
filebeat.inputs:
  - type: log
    enable: true
    tags: ["rsnapshot"]
    multiline.pattern: 'rsnapshot -c'
    multiline.negate: true
    multiline.match: after
    paths:
      - /path/rsnapshot/*

still looking for the correct multiline pattern to use.
Thanks for any help,
-f

2

Hey @rosseba,

At the moment it is not possible to filter out lines and then apply the multiline options. There is an open issue requesting that https://github.com/elastic/beats/issues/12562.

Would it be an option for you to filter out messages, but not joining the start and completion lines together?

3

I thought was a my config mistake!

This config:

  - type: log
    enable: true
    tags: ["rsnapshot"]
    include_lines: ['\/bin\/rsnapshot -c']
    paths:
      - /sftpusers/chroot/rsnapshot/*

filter just the two interested lines as separate entries, no single multiline entry, as you confirmed.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.