Greedymultine works in simulation but fails when actually used

Elastic Stack Kibana
1

Hello all,

I am trying to use the GREEDYMULTILINE in one of ingest pipeline as the GREEDYDATA is not working. So when I do the simulation using my grok processor in dev tools it works just fine but it fails when I actually try to use it. My filebeat logs says Provided Grok expressions do not match field value.
My simulation pipeline looks like this:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
{
  "grok" : {
    "field" : "my-message",
    "patterns" : [ "%{DATESTAMP:timestamp},%{NUMBER:counter}%{SPACE}tid:%{NOTSPACE:tracking_id}%{SPACE}%{DATA:LogLevel}%{SPACE}\\[%{JAVACLASS:ClassName}\\]%{SPACE}%{GREEDYMULTILINE:details}" ],
    "pattern_definitions" : {
      "GREEDYMULTILINE" : "(.|\n)*"
    }
  }
}
    ]
  },
  "docs": [
    {
      "_index": "generic-index",
      "_source": {
        "my-message": "2022-03-31 12:35:03,168 tid:123fy-th7 ERROR [org.id.web.profiles.support] Exception occurred during request processing\njava.lang.ClassCastException: class org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl cannot be cast to class org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument (org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl and org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument are in unnamed module of loader 'app')\n    at org.sourceid.saml20.profiles.idp.HandleAuthnRequest.getSetRequestedUserId(HandleAuthnRequest.java:228) ~[pf-protocolengine.jar:?]]"
      }
    }
  ]
}

And the output for this looks like:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "generic-index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "ClassName" : "org.id.web.profiles.support",
          "details" : """Exception occurred during request processing
java.lang.ClassCastException: class org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl cannot be cast to class org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument (org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl and org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument are in unnamed module of loader 'app')
    at org.sourceid.saml20.profiles.idp.HandleAuthnRequest.getSetRequestedUserId(HandleAuthnRequest.java:228) ~[pf-protocolengine.jar:?]]""",
          "my-message" : """2022-03-31 12:35:03,168 tid:123fy-th7 ERROR [org.id.web.profiles.support] Exception occurred during request processing
java.lang.ClassCastException: class org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl cannot be cast to class org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument (org.sourceid.saml20.xmlbinding.protocol.impl.LogoutRequestDocumentImpl and org.sourceid.saml20.xmlbinding.protocol.AuthnRequestDocument are in unnamed module of loader 'app')
    at org.sourceid.saml20.profiles.idp.HandleAuthnRequest.getSetRequestedUserId(HandleAuthnRequest.java:228) ~[pf-protocolengine.jar:?]]""",
          "counter" : "168",
          "tracking_id" : "123fy-th7",
          "LogLevel" : "ERROR",
          "timestamp" : "22-03-31 12:35:03"
        },
        "_ingest" : {
          "timestamp" : "2022-04-01T17:55:17.232978808Z"
        }
      }
    }
  ]
}

I also tried to use the multiline filter in my filebeat like:

  multiline.type: pattern
  #multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2})'
  #multiline.pattern: '^\n([0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2})'
  multiline.pattern: '^\[0-9]{4}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

But it still doesn't work. The second and third pattern works successfully in Go Playground - The Go Programming Language
I am not sure what am I doing wrong here.
Please help.
Apologies for the long explanation.

2

I was able to resolve this. I was using filestream input and it has a different multiline syntax than the log input.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.