Multiline with Ingest Node

[Fei_Yao]

I have the following log from Spring Boot, just for the demonstration purpose.

//log starts

2017-10-19 11:21:20.949 ERROR 29648 --- [http-nio-8080-exec-5] com.example.demo.GlobalExceptionHandler : Unhandled exception occurred

java.lang.IllegalArgumentException: The name params missing, sorry.
at com.example.demo.HiController.greeting(HiController.java:30) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]

//log ends

According to https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html. I've set up the following config in filebeat.yml

//config start

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

//config end

It will work fine if I just dump it into Elasticsearch as a message. What if I'm using Ingest Pipeline? The problem surfaces since the event doesn't match the grok pattern (SpringBoot Logging pattern). My guess is multiline is working from the filebeat's perspective, but when the event arrives Ingest Pipeline it failed there since it just have the message body and missing TIMESTAMP_ISO8601, etc.

//In Elasticsearch, I've setup the following pipeline
PUT _ingest/pipeline/springboot_v1
{
"description": "springboot-v1",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{TIMESTAMP_ISO8601:logtime}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{NUMBER:pid}%{SPACE}---%{SPACE}%{THREAD:thread}%{SPACE}%{LOGGERNAME:loggername}%{SPACE}:%{SPACE}%{GREEDYDATA:logmessage}"],
"pattern_definitions": {
"THREAD" : "\[([A-Za-z0-9-]| )+\]",
"LOGGERNAME" : "\S+"
}
}
}
]
}

So, I'm look for a solution. Please advice!

Thanks

[Fei_Yao]

@spinscale I noticed that you have a comment in other thread at Ingest Node, Multiline, Glassfish Logs

Would you please take a look of mine? Which is a very similar situation.

Thanks

[Fei_Yao]

For my log example:

//log starts

2017-10-19 11:21:20.949 ERROR 29648 --- [http-nio-8080-exec-5] com.example.demo.GlobalExceptionHandler : Unhandled exception occurred

java.lang.IllegalArgumentException: The name params missing, sorry.
at com.example.demo.HiController.greeting(HiController.java:30) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]

//log ends

This one log event seems to be broken into 2 events,

Event1:
2017-10-19 11:21:20.949 ERROR 29648 --- [http-nio-8080-exec-5] com.example.demo.GlobalExceptionHandler : Unhandled exception occurred
Event2:
java.lang.IllegalArgumentException: The name params missing, sorry.
at com.example.demo.HiController.greeting(HiController.java:30) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]

Since Event1 matches the pipeline Grok pattern it will work, where the Event2 doesn't match with the pattern and it will fail. To make the matter worse, the Filebeat's output.elasticsearch will stall as well because this event cannot be pushed through.

Any thoughts?

[steffens (Steffen Siering)]

Please use the </> button for logs and configuration files in order to preserve indentation and not triggering other test formatting by accident. When working with multiline it's helpful to see the actual structure, not just the text.

[Fei_Yao]

Steffen,
Here is the log that Im interested about:

2017-10-19 10:13:52.272 ERROR 29678 --- [http-nio-8081-exec-2] com.example.demo.GlobalExceptionHandler  : Unhandled exception occurred

java.lang.IllegalArgumentException: The name params cannot be empty.
	at com.example.demo.HiController.greeting(HiController.java:30) ~[classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]

Thanks

[steffens (Steffen Siering)]

If all events start with a timestamp, apply the multiline pattern to the timestamp and use negate: true. I think I've shown the pattern to use in another duplicate topic of yours.