Ingest Node, Multiline, Glassfish Logs

[iamkelvinloke (Kelvin Loke)]

This is purely a sharing thread that I hope I can help the beginner like me when they need to figure how to make Multiline works in Ingest Node, for Glassfish logs.

PUT _ingest/pipeline/test
{
    "description": "test",
    "on_failure": [
      {
        "set": {
          "field": "error",
          "value": "{{ _ingest.on_failure_message }}"
        }
      }
    ],
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "\\[%{TIMESTAMP_ISO8601:timestamp}\\] \\[%{DATA:server_version}\\] \\[%{LOGLEVEL:loglevel}\\] \\[%{DATA:message_id}\\] \\[%{DATA:class}\\] \\[%{DATA:thread}\\] \\[timeMillis: %{DATA:timemillis}\\] \\[%{DATA:levelvalue}\\] \\[\\[\\n\\s*%{GREEDYMULTILINE:message_detail}\\]\\]$"
          ],
          "pattern_definitions" : {
             "GREEDYMULTILINE" : "(.|\n)*"
          },
          "ignore_missing": true
        }
      }
    ]
}

I am no way an experience grok master here, so suggestion is welcome if there is room for improvement for above grok expression

Cheers!

[spinscale (Alexander Reelsen)]

Hey,

can you explain what exactly you want to achieve with the above configuration? If the ingest node receives a document, you will already have multiple lines of text in there or you will not (which means that every line is delivered as separate documents). In the first case everything is as expected, where as in the second case, you will still index one document per line, which is not what is intended. The component sending the logs to ES is responsible for correctly handling multiline messages, i.e. a filebeat

Hops this helps, otherwise I am happy to have misread your post and feel free to correct me

--Alex