Grokparsefailure when filtering a multiline json value

Elastic Stack Logstash
1

Due to memory limitation I cannot deploy Beats on my device. Therefore I pull metric data with a remote Http_poller into Logstash.
The response data of the poll-request looks like this:

{
	"memory": {
		"raw": "              total        used        free      shared  buff/cache   available\r\nMem:        6876856     4778664     1338016        4084      760176     2009568\r\nSwap:       2097148       24332     2072816"
	}
}

As you can see it is the plain output of the free command (Linux). If I do not use a filter and just print it to stdout with rubydebug it looks like this:

{
    "http_poller_metadata" => {
        ...truncated...
    },
    "@timestamp" => 2019-10-23T06:52:02.034Z,
    "@version" => "1",
    "memory" => {
        "raw" => "              total        used        free      shared  buff/cache   available\r\nMem:        6876856     4778664     1338016        4084      760176     2009568\r\nSwap:       2097148       24332     2072816"
    }
}

Now I want to put those numbers into fields using grok.
My Logstash config approach is this:

input {
  http_poller {
    urls => {
      test1 => "http://10.10.1.XXX/raw_metrics.json"
    }
    request_timeout => 60
    schedule => { "every" => "1m" }
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}
filter {
        grok {
                match => { '[memory][raw]' => '              total        used        free      shared  buff/cache   available\\r\\nMem:        %{NUMBER:total_memory}     %{NUMBER:used_memory}     %{NUMBER:free_memory}        %{NUMBER:shared_memory}      %{NUMBER:buffered_memory}     %{NUMBER:available_memory}\\r\\nSwap:       %{NUMBER:total_swap_memory}       %{NUMBER:used_swap_memory}     %{NUMBER:free_swap_memory}' }
        }
}
output {
        stdout { codec => rubydebug }
}

But this produces a grokparsefailure:

"tags" => [
        [0] "_grokparsefailure"
]

Do you have any hint for me? The grok debugger in Kabana showed no issues when using only the raw data against the match pattern.

Thank you!
Andreas

2

The string probably does not literally contain \r and \n, it probably contains a carriage return and newline. Try matching against two patterns, one for each line.

3

My issue was indeed a newline issue. I now split it now first with mutate->split and it works for me now :slight_smile:

It seems that the match just affected one line all the time.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.