Filebeat multiline problem with logs containing backticks

Elastic Stack Beats
1

I am trying to get multline logs into filebeat from an ASP.Net MVC solution. I noticed a problem when testing out the multiline pattern due to the logs contain ``` backtick characters, but hoped it was just due to the multiline test debugger. It turns out there actually is a problem. Anyone knows what to do with logs that contain backticks?

The error messages I get are Provided Grok expressions do not match field value

Sample log with several backticks:

System.Web.HttpException (0x80004005): A public action method 'Post' was not found on controller 'BABADOFAR.Controllers.LoginPageController'.
  at System.Web.Mvc.Controller.HandleUnknownAction(String actionName)
 at EPiServer.Web.Mvc.ActionControllerBase.HandleUnknownAction(String actionName)
at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult  asyncResult, ExecuteCoreState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegat e(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
 at  System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute   ()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
2

Filebeat doesn't use grok so perhaps this is a Logstash error and this should be a post in the Logstash topic.

However Filebeat can do multiline on its own. See

3

Thanks for your help but I'm sorry, I am actually using filebeat.
This is the multiline pattern I'm using.

multiline.match: after
multiline.negate: true
multiline.pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
4

this multiline config just looks for a date at the beginning of the line. Otherwise filebeat doesn't care about the content besides the final new-line characters. Content is send to logstash as is.

5

I'm using elasticsearch pipeline, though, not logstash. The lines are split where the backtick character occurs in the logs. I will register this as a bug then.

6

Do you have a more 'complete' log with timestamps and such? Maybe share more details of your filebeat and ingest node pipeline config? Have you checked if filebeat is splitting the event on the backtick (e.g. output to file or stdout)?

With the amount of information we've so far I can not tell if it's a bug in beats or ingest node.

7

I tried now without using the ingest pipeline, and I get the complete messages, new lines, weird characters and all. So there is most likely something wrong in my grok. Sorry about the fuzz!

8

Thanks for reporting back. Would still be interesting to learn/investigate if this might be a potential bug in ES or not. I'd recommend the ingest node simulate API with failing cases.

9

I finally found the problem. It was a misconfiguration in the filebeat.yml prospectors section. I had two log prospectors, but only one multiline section, somehow I was under the impression that the multiline settings would apply to all prospectors. Yet another PEBKAC :slight_smile:

10

Phew... no bug. Thanks for reporting back.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.