# Logstash Filter issue

**URL:** <https://discuss.elastic.co/t/logstash-filter-issue/300950>\
**Category:** Logstash\
**Created:** [March 29, 2022, 11:06am UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950 "2022-03-29T11:06:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\
**Post date:** [March 29, 2022, 11:06am UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/1 "2022-03-29T11:06:54Z")

</div>

Hi,

I have a problem with a logstash filter.

The message which has to be filtered looks like:

> Imported data: {"total":10,"valid":7,"violations":{"missing":["359072065634251","359072065633741"],"Data":["359072065634251","359072065633741","359072065634863"]}}

and my filter is:

```auto
if "Imported data:" in [message] {
      mutate {
        add_field => { "test" => "%%{message}" }
      }
      mutate {
        gsub => ["test", "^.* ", ", "]
      }
    }

```

This leads to:

> imb.total:10  
> imb.valid:7  
> test.violations.missing:["359072065634251","359072065633741","359072065634863"]  
> test.violations.data:["359072065634251","359072065633741"]

So far so good, the problem is that NewRelic can't handle the two arrays (missing and data) and apparently expects a blank after the comma.

Does anyone have an idea how I have to change the filter so that the arrays contain the following:

> test.violations.missing:["359072065634251", "359072065633741", "359072065634863"]  
> test.violations.data:["359072065634251", "359072065633741"]

Regards  
Thorsten

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 29, 2022, 3:36pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/2 "2022-03-29T15:36:21Z")

</div>

You could try using mutate+gsub to replace `","` with `", "`.

---

<div class="post-metadata">

**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\
**Post date:** [March 30, 2022, 1:32pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/3 "2022-03-30T13:32:07Z")

</div>

Hi Badger,

you mean I should change the filter like this:

```auto
if "Imported data:" in [message] {
      mutate {
        add_field => { "test" => "%%{message}" }
      }
      mutate {
        gsub => ["test", "^.* ", "",",",", "]
      }
    }

```

Regards  
Thorsten

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 30, 2022, 5:18pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/4 "2022-03-30T17:18:13Z")

</div>

> [@cRaZyT](#):
>
> `gsub => ["test", "^.* ", "",",",", "]`

You need to provide the field name in the second triplet.

```
gsub => ["test", "^.* ", "", "test", ",", ", "]

```

---

<div class="post-metadata">

**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\
**Post date:** [April 4, 2022, 4:06pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/5 "2022-04-04T16:06:26Z")

</div>

It's not working, NewRelic won't count the length of both arrays.

Is there maybe a way to get the array length with logstash and add another field by enhancing the following filter?

```auto
if "Imported data:" in [message] {
      mutate {
        add_field => { "test" => "%%{message}" }
      }
      mutate {
        gsub => ["test", "^.* ", ", "]
      }
    }

```

That the output leads to:

> test.total:10  
> test.valid:7  
> test.violations.missing:["359072065634251","359072065633741","359072065634863"]  
> test.violations.data:["359072065634251","359072065633741"]  
> test.violations.missing.total: 3  
> test.violations.data.total: 2

Regards  
Thorsten

---

<div class="post-metadata">

**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\
**Post date:** [April 7, 2022, 1:49pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/6 "2022-04-07T13:49:11Z")

</div>

I tried it with the following code without success

```auto
if "Imported data:" in [message] {
  mutate {
    add_field => { "test" => "%%{message}" }
  }
  mutate {
    gsub => ["test", "^.* ", ", "]
  }
  ruby { code => "event['test.violations.missing.total'] = event['test.violations.missing"].length" }
  ruby { code => "event['test.violations.data.total'] = event['test.violations.data"].length" }
}

```

Here I get neither the two new fields nor an error message, what could be the reason?

Regards  
Thorsten

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 5, 2022, 1:49pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/300950/7 "2022-05-05T13:49:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.