Manage multiline

[Fei_Yao]

Hi,
I've read https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html. But It didn't help me for the downstream Elasticsearch Ingest Pipeline Grok Processor.

The challenge I have is the following ONE log event has been broken into TWO events by Filebeat. The first one has great Grok pattern I'd like to match. But the second one doesn't have such pattern to match and it breaks the pipeline.

I wonder Filebeat team or Elasticsearch team has any solution for this? Ideally, I want Filebeat be able to 1) process multiline, which it does well, 2) make sure below is one event. Possible?

2017-10-19 22:48:08.166 ERROR 19345 --- [http-nio-8080-exec-1] com.example.demo.GlobalExceptionHandler  : Unhandled exception occurred

java.lang.IllegalArgumentException: Missing parameter
	at com.example.demo.HiController.greeting(HiController.java:30) ~[classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]

[steffens (Steffen Siering)]

Looks like you want to apply multiline on the date pattern (e.g. '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} ') and use negate true.

You can configure multiple grok patterns in your Ingest Node Pipeline. The grok processor will try each definition (one after another) and only fail if none matches.