Hello all,
I am trying to use the GREEDYMULTILINE in one of ingest pipeline as the GREEDYDATA is not working. So when I do the simulation using my grok processor in dev tools it works just fine but it fails when I actually try to use it. My filebeat logs says Provided Grok expressions do not match field value.
My simulation pipeline looks like this:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"grok" : {
"field" : "my-message",
"patterns" : [ "%{DATESTAMP:timestamp},%{NUMBER:counter}%{SPACE}tid:%{NOTSPACE:tracking_id}%{SPACE}%{DATA:LogLevel}%{SPACE}\\[%{JAVACLASS:ClassName}\\]%{SPACE}%{GREEDYMULTILINE:details}" ],
"pattern_definitions" : {
"GREEDYMULTILINE" : "(.|\n)*"
}
}
}
]
},
"docs": [
{
"_index": "generic-index",
"_source": {
"my-message": "2022-03-31 12:35:03,168 tid:123fy-th7 ERROR [org.id.web.profiles.support] Exception occurred during request processing\njava.lang.ClassCastException: class org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl cannot be cast to class org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument (org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl and org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument are in unnamed module of loader 'app')\n at org.sourceid.saml20.profiles.idp.HandleAuthnRequest.getSetRequestedUserId(HandleAuthnRequest.java:228) ~[pf-protocolengine.jar:?]]"
}
}
]
}
And the output for this looks like:
{
"docs" : [
{
"doc" : {
"_index" : "generic-index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"ClassName" : "org.id.web.profiles.support",
"details" : """Exception occurred during request processing
java.lang.ClassCastException: class org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl cannot be cast to class org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument (org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl and org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument are in unnamed module of loader 'app')
at org.sourceid.saml20.profiles.idp.HandleAuthnRequest.getSetRequestedUserId(HandleAuthnRequest.java:228) ~[pf-protocolengine.jar:?]]""",
"my-message" : """2022-03-31 12:35:03,168 tid:123fy-th7 ERROR [org.id.web.profiles.support] Exception occurred during request processing
java.lang.ClassCastException: class org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl cannot be cast to class org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument (org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl and org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument are in unnamed module of loader 'app')
at org.sourceid.saml20.profiles.idp.HandleAuthnRequest.getSetRequestedUserId(HandleAuthnRequest.java:228) ~[pf-protocolengine.jar:?]]""",
"counter" : "168",
"tracking_id" : "123fy-th7",
"LogLevel" : "ERROR",
"timestamp" : "22-03-31 12:35:03"
},
"_ingest" : {
"timestamp" : "2022-04-01T17:55:17.232978808Z"
}
}
}
]
}
I also tried to use the multiline filter in my filebeat like:
multiline.type: pattern
#multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2})'
#multiline.pattern: '^\n([0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2})'
multiline.pattern: '^\[0-9]{4}-[0-9]{2}'
multiline.negate: true
multiline.match: after
But it still doesn't work. The second and third pattern works successfully in Go Playground - The Go Programming Language
I am not sure what am I doing wrong here.
Please help.
Apologies for the long explanation.