Understanding Multiline

[Louuu (Louis Boyd)]

Hi,

I have started using filebeat recently and have been looking at the multiline configurations as shown in the documentation.

https://play.golang.org/p/LQpFDR34t9

According to that, my multiline regex works as expected.

However it is returning as multiple lines as shown below and ignoring the multiline pattern which I have applied

`:{"type":"radiusdetail"},"input_type":"log","message":"Thu Mar  9 00:50:48 2017","offset":2389,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tUser-Name = 'test@test.co.uk'","offset":2438,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tAcct-Status-Type = Start","offset":2464,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tAcct-Session-Id = '1184'","offset":2490,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tFramed-IP-Address = 192.168.105.1","offset":2525,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tNAS-IP-Address = 192.168.1.1","offset":2555,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tCalling-Station-Id = '192.168.1.1'","offset":2591,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tCalled-Station-Id = '192.168.1.1'","offset":2626,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tEvent-Timestamp = 'Mar  9 2017 00:50:48 GMT'","offset":2672,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tAcct-Unique-Session-Id = 'c7ef06b03b3786387596bf072db184d7'","offset":2733,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}
{"@timestamp":"2017-03-09T00:50:49.671Z","beat":{"hostname":"app.radius.local","name":"app.radius.local","version":"5.2.2"},"fields":{"type":"radiusdetail"},"input_type":"log","message":"\tTimestamp = 1489020648","offset":2757,"source":"/var/log/radius/radacct/10.122.217.4/detail-20170309","type":"log"}`

Below is my configuration file for filebeat

filebeat.prospectors:

- input_type: log


  paths:
    - /var/log/radius/radacct/*/detail-*

multiline.pattern: '^\t+.*$'
multiline.negate: false
multiline.match: after
   
fields:
   type: radiusdetail

output.file:
  path: "/tmp/filebeat"
  filename: filebeat

I really hope someone can help me with this!
Lou

[andrewkroh]

The indentation is incorrect. The multiline options are members of the prospector config so they need to be at the same level as input_type and paths.

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/radius/radacct/*/detail-*
  multiline.pattern: '^\t+.*$'
  multiline.negate: false
  multiline.match: after
  document_type: radiusdetail # You might want this option instead of using fields.

[Louuu (Louis Boyd)]

Hi Andrew,

I shall give this a try in my test environment when I am next working on this project!

Thank you!