Multiline plugin not merging lines and throwing _grokparsefailure

Elastic Stack Logstash
1

Hi,

In logstash 1.x, used multiline plugin to merge all lines those are not starting with Timestamp and found issue that in some cases it didn't worked.

Filter:

multiline {
patterns_dir => "/u01/data/conf/patterns"
pattern => "^%{SERVERTIMESTAMP} "
negate => true
what => "previous"
}
mutate {
gsub => ["message", "\n", " "]
gsub => ["message", "\t", " "]
}
grok {
break_on_match => false
patterns_dir => "/u01/data/conf/patterns"
match => ["message", "%{IDCS_LOG_FM}"]
}

Tested in logstash 5.x, but not able to reproduce this issue.

Need help?

2

The multiline filter has been deprecated, so use the multiline functionality of Filebeat or the multiline codec plugin instead.

3

Yes u r right. Using codec multiline in input it's working fine with logstash 5.x

But this issue is in logstash 1.x where multiline plugin was present.

4

Below are the examples:

Good Case (All lines merged)
Dec 22, 2017 6:08:10 AM oracle.idaas.common.event.impl.EventManagerImpl logEvent INFO: Event: system.info.runtime, Generated: Fri Dec 22 06:08:10 GMT 2017, Host: idcssydenydyntopob1746try1-ssov1-1.idcs.oraclecloud.internal (10.196.132.46), with targets metrics: false, audit: false, notification: false, analyze: false, subscribe: false oracle.idaas.common.cache.RemoteAPIProxySCIMService RemoteAPIProxySCIMService/getCompositeKeyBasedSearch: entered with resourceType:Key tenant:idcs-cloudinfra-aucom-east-1 filter:AND{left=AND{left=FilterImpl{propertyName=keyAlias, op=EQUALS, propertyValue=GenericCGForLBaaS_APPID_next}, isComplex=false, right=FilterImpl{propertyName=keyStoreId, op=EQUALS, propertyValue=2019b30136a543a39d2b61860a170593}}, isComplex=false, right=FilterImpl{propertyName=type, op=EQUALS, propertyValue=Key}} ECID: eOvGZ0nwZ00000000, Tenant: idcs-f5b6c873e93b4d4c99083f23ac2eb40a, RID: 0:1:6:2, Service Name: SSO Actor ID: null, Actor Name: null, Actor Display Name: null, Actor Type: null, Client IP: 10.245.54.162, Client ID: null, Client Name: null, Session Id: null Event Data: {"idcs_ext_url":"https://%tenant%.identity.c9qa132.oc9qadev.com"}

Bad Case (Multiline didn't merged)
Dec 22, 2017 6:24:26 AM oracle.idaas.common.event.impl.EventManagerImpl logEvent

INFO: Event: system.info.runtime, Generated: Fri Dec 22 06:12:30 GMT 2017, Host: idcssydenydyntopob1746try1-ssov1-1.idcs.oraclecloud.internal (10.196.132.46), with targets metrics: false, audit: false, notification: false, analyze: false, subscribe: false oracle.idaas.common.cache.RemoteAPIProxySCIMService RemoteAPIProxySCIMService/get: entered with resourceType:GlobalConfig tenant:idcs-cloudinfra-aucom-east-1 options:SearchOptionsImpl{sortOrder=ASCENDING, count=NumberValueImpl{number=50}, startIndex=NumberValueImpl{number=1}} id:GlobalConfig ECID: qY3sW1L0Z00000000, Tenant: idcs-eb5b11d44630441e9d76b336d51f8615, RID: 0:1, Service Name: SSO Actor ID: null, Actor Name: null, Actor Display Name: null, Actor Type: null, Client IP: 10.244.186.113, Client ID: null, Client Name: null, Session Id: null Event Data: {"idcs_ext_url":"https://%tenant%.identity.c9qa132.oc9qadev.com"}

FINE: Service [ sso-v1::10.196.132.46 ] Service Status :: RUNNING

5

I would recommend that you upgrade from Logstash 1.x. There were good reasons why the multiline filter plugin was deprecated.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.