# Multiline pattern not working for more complex cases **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 16, 2021, 6:07pm UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407 "2021-03-16T18:07:03Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![RFeiten](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@RFeiten](https://discuss.elastic.co/u/RFeiten)\ **Post date:** [March 16, 2021, 6:07pm UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407/1 "2021-03-16T18:07:03Z")
Hi all! I'm facing an issue while collecting logs using Filebeat (7.x) in conjunction to a pipeline in Logstash. The multi-line scenario has been overcome by checking elastic documentation, however, there are specific scenarios where error stack traces are being generated with additional information in our logs, thus, not respecting the regex pattern used, and then lots of "\_grokParseFailures" are happening in Logstash, during ingestion process. Below, we have a sample of how log sample looks like ```auto 2021-03-10T00:27:21.0691085+00:00 0HM730JRVJI5F:0000001F [ERR] Something went wrong { "applicationName": "...", "id": null, "responseCode": 409, "responseDescription": "UNABLE_TO_LOCK_ROW" },https://.... at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) Headers: { Date: Wed, 10 Mar 2021 00:27:21 GMT Server: nginx Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 314 },Method: PATCH, RequestUri: ... { Transfer-Encoding: chunked Content-Type: application/json; charset=utf-8 } (32238518) 2021-03-10T00:27:21.0694640+00:00 0HM730JRVJI5F:0000001F [INF] Request finished in 6041.9749ms 409 application/json (791a596a) ``` This is how the multi-line pattern has been configured in filebeat configuration ```auto multiline.type: pattern multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after ``` We can see that my regex pattern just looks for spaces, which is fine for most of the cases, unless we have those scenarios where JSON data is being collect as well. Ideally, what I want to achieve is to only get a record when it starts with the timestamp information, and proceed with it until the next timestamp pattern (in the line beginning) is found. With the current implementation, this makes lines as "},https://...." (line 6) be interpreted as a new line, thus, adding the \_grokParseFailure at ingestion. How to overcome this situation? ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [March 16, 2021, 6:12pm UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407/2 "2021-03-16T18:12:52Z")
> [@RFeiten](#): > > Ideally, what I want to achieve is to only get a record when it starts with the timestamp information, and proceed with it until the next timestamp pattern (in the line beginning) is found. Have you tried adjusting the regex pattern to look for a datetime at the start of the line, so something like this? ```auto ^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2} ``` Shaunak ---
**Author:** ![RFeiten](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@RFeiten](https://discuss.elastic.co/u/RFeiten)\ **Post date:** [March 16, 2021, 8:32pm UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407/3 "2021-03-16T20:32:08Z")
Thanks for your reply, @shaunak, I tried your suggestion, but it become even worse that it was before. for a couple entries, that actually originated 43 entries, that was supposed to give me only 5 log entries. At the end, the beat configuration has been configured like ```auto multiline.type: pattern multiline.pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}' multiline.negate: false ``` Also, I'm afraid the pattern may not work, as we can have similar date pattern in JSON as well... ```auto 2021-03-10T00:27:21.0691085+00:00 0HM730JRVJI5F:0000001F [ERR] error message "applicationName": "", "id": null, "responseCode": 409, "serverResponseCode": null, "responseDateTime": "2021-03-10T00:27:21.046Z", "responseMessage": "...", "responseDescription": "..." } at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker) at Microsoft.AspNetCore.Routing.EndpointMiddleware.... 2021-03-10T00:27:21.0694640+00:00 0HM730JRVJI5F:0000001F [INF] info message ``` Any ideas? ---
**Author:** ![aj.pahl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aj.pahl/32/58989_2.png) [@aj.pahl](https://discuss.elastic.co/u/aj.pahl)\ **Post date:** [March 20, 2021, 3:04am UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407/4 "2021-03-20T03:04:08Z")
@RFeiten I think if you change multiline.negate to true and add "multiline.match: after" it should start to recognize multiline events. ---
**Author:** ![RFeiten](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@RFeiten](https://discuss.elastic.co/u/RFeiten)\ **Post date:** [March 22, 2021, 4:37pm UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407/5 "2021-03-22T16:37:52Z")
Thanks, that did the trick! ```auto multiline.type: pattern multiline.pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{7}\+' multiline.negate: true multiline.match: after ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 19, 2021, 6:38pm UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-for-more-complex-cases/267407/6 "2021-04-19T18:38:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.