Logstash Filter issue

Elastic Stack Logstash
1

Hi,

I have a problem with a logstash filter.

The message which has to be filtered looks like:

Imported data: {"total":10,"valid":7,"violations":{"missing":["359072065634251","359072065633741"],"Data":["359072065634251","359072065633741","359072065634863"]}}

and my filter is:

if "Imported data:" in [message] {
      mutate {
        add_field => { "test" => "%%{message}" }
      }
      mutate {
        gsub => [ "test", "^.* ", ", " ]
      }
    }

This leads to:

imb.total:10
imb.valid:7
test.violations.missing:["359072065634251","359072065633741","359072065634863"]
test.violations.data:["359072065634251","359072065633741"]

So far so good, the problem is that NewRelic can't handle the two arrays (missing and data) and apparently expects a blank after the comma.

Does anyone have an idea how I have to change the filter so that the arrays contain the following:

test.violations.missing:["359072065634251", "359072065633741", "359072065634863"]
test.violations.data:["359072065634251", "359072065633741"]

Regards
Thorsten

2

You could try using mutate+gsub to replace "," with ", ".

3

Hi Badger,

you mean I should change the filter like this:

if "Imported data:" in [message] {
      mutate {
        add_field => { "test" => "%%{message}" }
      }
      mutate {
        gsub => [ "test", "^.* ", "",",",", "]
      }
    }

Regards
Thorsten

4

You need to provide the field name in the second triplet.

gsub => [ "test", "^.* ", "",      "test", ",", ", "]
5

It's not working, NewRelic won't count the length of both arrays.

Is there maybe a way to get the array length with logstash and add another field by enhancing the following filter?

if "Imported data:" in [message] {
      mutate {
        add_field => { "test" => "%%{message}" }
      }
      mutate {
        gsub => [ "test", "^.* ", ", " ]
      }
    }

That the output leads to:

test.total:10
test.valid:7
test.violations.missing:["359072065634251","359072065633741","359072065634863"]
test.violations.data:["359072065634251","359072065633741"]
test.violations.missing.total: 3
test.violations.data.total: 2

Regards
Thorsten

6

I tried it with the following code without success

if "Imported data:" in [message] {
  mutate {
    add_field => { "test" => "%%{message}" }
  }
  mutate {
    gsub => [ "test", "^.* ", ", " ]
  }
  ruby { code => "event['test.violations.missing.total'] = event['test.violations.missing"].length" }
  ruby { code => "event['test.violations.data.total'] = event['test.violations.data"].length" }
}

Here I get neither the two new fields nor an error message, what could be the reason?

Regards
Thorsten