Filebeat multiline problem with logs containing backticks

(Christoffer Vig) #1

I am trying to get multline logs into filebeat from an ASP.Net MVC solution. I noticed a problem when testing out the multiline pattern due to the logs contain ``` backtick characters, but hoped it was just due to the multiline test debugger. It turns out there actually is a problem. Anyone knows what to do with logs that contain backticks?

The error messages I get are Provided Grok expressions do not match field value

Sample log with several backticks:

System.Web.HttpException (0x80004005): A public action method 'Post' was not found on controller 'BABADOFAR.Controllers.LoginPageController'.
  at System.Web.Mvc.Controller.HandleUnknownAction(String actionName)
 at EPiServer.Web.Mvc.ActionControllerBase.HandleUnknownAction(String actionName)
at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult  asyncResult, ExecuteCoreState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegat e(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
 at  System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute   ()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

(Andrew Kroh) #2

Filebeat doesn't use grok so perhaps this is a Logstash error and this should be a post in the Logstash topic.

However Filebeat can do multiline on its own. See

(Christoffer Vig) #3

Thanks for your help but I'm sorry, I am actually using filebeat.
This is the multiline pattern I'm using.

multiline.match: after
multiline.negate: true
multiline.pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}"

(Steffen Siering) #4

this multiline config just looks for a date at the beginning of the line. Otherwise filebeat doesn't care about the content besides the final new-line characters. Content is send to logstash as is.

(Christoffer Vig) #5

I'm using elasticsearch pipeline, though, not logstash. The lines are split where the backtick character occurs in the logs. I will register this as a bug then.

(Steffen Siering) #6

Do you have a more 'complete' log with timestamps and such? Maybe share more details of your filebeat and ingest node pipeline config? Have you checked if filebeat is splitting the event on the backtick (e.g. output to file or stdout)?

With the amount of information we've so far I can not tell if it's a bug in beats or ingest node.

(Christoffer Vig) #7

I tried now without using the ingest pipeline, and I get the complete messages, new lines, weird characters and all. So there is most likely something wrong in my grok. Sorry about the fuzz!

(Steffen Siering) #8

Thanks for reporting back. Would still be interesting to learn/investigate if this might be a potential bug in ES or not. I'd recommend the ingest node simulate API with failing cases.

(Christoffer Vig) #9

I finally found the problem. It was a misconfiguration in the filebeat.yml prospectors section. I had two log prospectors, but only one multiline section, somehow I was under the impression that the multiline settings would apply to all prospectors. Yet another PEBKAC :slight_smile:

(Steffen Siering) #10

Phew... no bug. Thanks for reporting back.

(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.