Filebeats causing \t not to be caught in multiline

[ksnapes]

Hi,

I'm having an issue when sending logs from filebeats to logstash

I am trying to capture java stack trace events in tomcat log files. I am using multiline to capture the events that span multiple lines and it is working for all events except for those similar to the event below.

The problem i have with these events is that all lines starting with tabs (\t) are not being added to the event and are being sent as a seperate event. e.g. "\tat javax.script.ScriptEngine$eval.call(Unknown Source)"

As I am unable to have logstash receive from filebeat and parse multiline I am wondering if this is a known behaviour with filebeat not handling \t tabs well.

I have tried this configuration as well as one matching on the timestamp
multiline.pattern: '^[[:space:]]+(at|.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after

2014-12-19 15:07:55,408 http-bio-8443-exec-24 ERROR user1 907x93872x2 11924rz 10.34.89.41 /secure/AjaxIssueEditAction!default.jspa [onresolve.jira.groovy.GroovyCustomField] javax.script.ScriptException: java.lang.NullPointerException: Cannot get property 'value' on null object
javax.script.ScriptException: javax.script.ScriptException: java.lang.NullPointerException: Cannot get property 'value' on null object
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:117)
at javax.script.AbstractScriptEngine.eval(Unknown Source)
at javax.script.ScriptEngine$eval.call(Unknown Source)
at com.onresolve.jira.groovy.GroovyCustomField.getValueFromIssue(GroovyCustomField.groovy:160)
at com.atlassian.jira.issue.fields.CustomFieldImpl.getCustomFieldParamsFromIssue(CustomFieldImpl.java:1474)
at com.atlassian.jira.issue.fields.CustomFieldImpl.populateFromIssue(CustomFieldImpl.java:693)
at com.atlassian.jira.issue.fields.screen.FieldScreenRenderLayoutItemImpl.populateFromIssue(FieldScreenRenderLayoutItemImpl.java:103)
at com.atlassian.jira.issue.fields.rest.FieldHtmlFactoryImpl.getEditFields(FieldHtmlFactoryImpl.java:135)
at sun.reflect.GeneratedMethodAccessor2143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:129)
at com.sun.proxy.$Proxy299.getEditFields(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)

[ruflin]

Normally the issue here is related to the regexp. Unfortunately regexp in Filebeat and LS are a bit different. Also it depends on the version of Filebeat you are using.

I recommend you to use our regexp playground here (https://www.elastic.co/guide/en/beats/filebeat/6.2/_test_your_regexp_pattern_for_multiline.html) to test your regexp agains the specific message. Note: This link assumes you use Filebeat 6.x.

[ksnapes]

I'm using filebeat 6.2.1 and sending to logstash 6.2.1

I did read somewhere that filebeat escapes tabs when sending them to logstash and was wondering it that could be causing an issue with the multiline.

I'll give the the playground a go and report back

[ksnapes]

https://play.golang.org/p/f5jUaJMDBl8

Even thought the regex im using is simple it seems to work correctly in the regex playground.
Yet with the same config filebeats will still split each line starting with a tab into a new line and escape it with \t.

Filebeats doesn't have any preprocessing functions to remove white space does it?

[ruflin]

Sorry for the late answer. Filebeat does not have preprocessing in place and when the multiline is applied on the Filebeat side, no escaping is applied.

It's odd that it's working in the playground but not Filebeat. Can you share how you put the sequence into the filebeat config?

[ksnapes]

Sure, this is the sequence

multiline.pattern: `^\d{4}-\d{2}-\d{2}`
multiline.negate: true
mutliline.match: after

filebeat.prospectors:
-type:log
paths:
/opt/tomcat/logs/catalina.out

output.console:
   pretty: true

[ruflin]

In the above there isn't any \t in the multiline config?

[ksnapes]

Sorry, I dont follow.
There isn't any \t in the multiline config i use in the regex playground either.

How should the \t be used in the multiline config?
I just assumed it would be captured in the multiline event because it didn't start with a date

[steffens (Steffen Siering)]

Did you post your complete configuration file as is? With all indentation? The configuration looks pretty wrong as is.

filebeat doesn't interpret the tab symbol. But multiline has a timeout. Some loggers unfortunately don't have a flush timeout (log4j with RollingFileAppender?), but only flush once the buffer is full.

Using this configuration, you still get split events?

filebeat.prospectors:
- type:log
  paths:
    - /opt/tomcat/logs/catalina.out

  multiline.pattern: '^\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  mutliline.match: after
  multiline.timeout: 0

output.console:
   pretty: true

Note: due to multiline and untimed flush timeout, filebeat can get 'stalled' until the start of the next event, due to the logging processing not behaving well.

[ksnapes]

This is really silly of me but it seems like i didnt have the indentation right on my multiline statements.

I'm able to get a whole java stacktrace sent as one event now,

Thanks a bunch for the help, hopefully anyone else with this problem sees this and can get a quick fix