Filebeat Multiline Messages Ingest Pipeline

Elastic Stack Beats
1

I need some help getting a multiline message from Filebeat Autodiscover ingested into ElasticSearch via an Ingest Pipeline.

The messages are not correctly rendering in Kibana Discover, example:

image
image1108×115 15.7 KB

The raw message (taken from the Filebeat logs in Debug mode) and shown here being used in Postman to simulate the pipeline looks like this:

image
image772×173 8.77 KB

The result of the Simulate action results in this:

image
image736×174 8.61 KB

Notice the double-backslash "\n"s in the output.

This is the GROK pattern I'm using (I got the GREEDYMULTILINE pattern from another question), this particular case is matching on the last pattern in the array:

"grok": {
            "field": "message",
            "patterns": [
                "%{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{CUSTOM_LEVEL} %{FUNCTION_NAME} %{JSON_CONTENT:TempJsonData}",
                "%{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{CUSTOM_LEVEL} %{JSON_CONTENT:TempJsonData}",
                "%{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{CUSTOM_LEVEL} %{GREEDYDATAMULTILINE:TempMessage}",
                "%{GREEDYDATAMULTILINE:TempMessage}"
            ],
            "pattern_definitions": {
                "CUSTOM_LEVEL": "(\\[)(%{DATA:TempLevel})(\\])",
                "FUNCTION_NAME": "(\\[function:)(%{DATA:FunctionName})(\\])",
                "JSON_CONTENT": "\\{.*\\}",
                "GREEDYDATAMULTILINE": "(.|\n)*"
            },
            "trace_match": true
        }

Can anyone help with this?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.