# Problem : Logstash filtering mutate/grok

**URL:** <https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456>\
**Category:** Logstash\
**Created:** [February 7, 2022, 11:34am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456 "2022-02-07T11:34:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![personal.riz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/personal.riz/32/98989_2.png) [@personal.riz](https://discuss.elastic.co/u/personal.riz)\
**Post date:** [February 7, 2022, 11:34am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/1 "2022-02-07T11:34:55Z")

</div>

Hi guys, iam new in elasticstack environment and i have some question below

i have a problem with logstash filter parsing, i just config my firewall labs to sending json log to logstash using tcp port 5000.

how to use filter mutate/grok to separate this `"LogId", "NodeId", "Facility", "Type", "Event", "Action", "Src", "Dst", "Service", "Protocol", "Sport", "Dport", "RuleId", "NatSrc", "NatDst", "NatSport", "NatDport", "Srcif", "SrcVlan", "CompId", "NatRuleId", "SenderType", "Situation", "EventId"` from field "message" ???

and to create new field `"LogId", "NodeId", "Facility", "Type", "Event", "Action", "Src", "Dst", "Service", "Protocol", "Sport", "Dport", "RuleId", "NatSrc", "NatDst", "NatSport", "NatDport", "Srcif", "SrcVlan", "CompId", "NatRuleId", "SenderType", "Situation", "EventId"`

some information :

```auto
Table
@timestamp Feb 7, 2022 @ 17:56:39.752
	
@version 1
	
_id IyXU034BJO1ry1ijGUSx
	
_index ngfw
	
_score - 
	
_type _doc
	
host xxx.xxx.xxx.xxx
	
message
<6>{"Timestamp":"2022-02-07 17:56:39", "LogId":"876936284", "NodeId":"xxx.xxx.xxx.xxx", "Facility":"Inspection", "Type":"Notification", "Event":"New connection", "Action":"Allow", "Src":"xxx.xxx.xxx.xxx", "Dst":"xxx.xxx.xxx.xxx", "Service":"TCP/30303", "Protocol":"6", "Sport":"39118", "Dport":"30303", "RuleId":"109.8", "NatSrc":"xxx.xxx.xxx.xxx", "NatDst":"xxx.xxx.xxx.xxx", "NatSport":"56458", "NatDport":"30303", "Srcif":"1", "SrcVlan":"3", "CompId":"CANAJKT-NGFW node 1", "NatRuleId":"260100.0 (NAT definition from engine)", "ReceptionTime":"2022-02-07 17:56:16", "SenderType":"Firewall", "Situation":"Connection_Allowed", "EventId":"6896406339135907837"}
	
port 53,824
	
tags _mutate_error

```

===============================================================================

```auto
JSON
{
  "_index": "ngfw",
  "_type": "_doc",
  "_id": "IyXU034BJO1ry1ijGUSx",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2022-02-07T10:56:39.752Z",
    "host": "xxx.xxx.xxx.xxx",
    "tags": [
      "_mutate_error"
    ],
    "port": 53824,
    "message": [
      "<6>{\"Timestamp\":\"2022-02-07 17:56:39\"",
      "\"LogId\":\"876936284\"",
      "\"NodeId\":\"xxx.xxx.xxx.xxx\"",
      "\"Facility\":\"Inspection\"",
      "\"Type\":\"Notification\"",
      "\"Event\":\"New connection\"",
      "\"Action\":\"Allow\"",
      "\"Src\":\"xxx.xxx.xxx.xxx\"",
      "\"Dst\":\"xxx.xxx.xxx.xxx\"",
      "\"Service\":\"TCP/30303\"",
      "\"Protocol\":\"6\"",
      "\"Sport\":\"39118\"",
      "\"Dport\":\"30303\"",
      "\"RuleId\":\"109.8\"",
      "\"NatSrc\":\"xxx.xxx.xxx.xxx\"",
      "\"NatDst\":\"xxx.xxx.xxx.xxx\"",
      "\"NatSport\":\"56458\"",
      "\"NatDport\":\"30303\"",
      "\"Srcif\":\"1\"",
      "\"SrcVlan\":\"3\"",
      "\"CompId\":\"CANAJKT-NGFW node 1\"",
      "\"NatRuleId\":\"260100.0 (NAT definition from engine)\"",
      "\"ReceptionTime\":\"2022-02-07 17:56:16\"",
      "\"SenderType\":\"Firewall\"",
      "\"Situation\":\"Connection_Allowed\"",
      "\"EventId\":\"6896406339135907837\"}"
    ],
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2022-02-07T10:56:39.752Z"
    ]
  },
  "sort": [
    1644231399752
  ]
}

```

===============================================================================

here is my logstash configuration : ngfw.conf

```auto
input {
  tcp {
    port => 5000
  }
}

filter {
  mutate {
       split => {"message" => ","}
       remove_field => ["message", "@version"]
  }
}

output {
  elasticsearch{
    hosts => "100.100.100.100:9200"
    index => "ngfw"
  }
}

```

Maybe you have best advice for me, thankyou

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 7, 2022, 6:21pm UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/2 "2022-02-07T18:21:06Z")

</div>

Please format your text using markdown. At the moment it is impossible to tell whether your message field is an array of strings, or JSON, or something else.

Select the logstash configuration and click \</\> in the toolbar above the edit pane. Note that the preview pane will change from

input {  
tcp {  
port =\> 5000  
}  
}

to

```auto
input {
    tcp {
        port => 5000
    }
}

```

then do the same for the JSON.

---

<div class="post-metadata">

**Author:** ![personal.riz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/personal.riz/32/98989_2.png) [@personal.riz](https://discuss.elastic.co/u/personal.riz)\
**Post date:** [February 8, 2022, 1:54am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/3 "2022-02-08T01:54:20Z")

</div>

sorry, i forgot about that. I have updated the post

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 8, 2022, 2:15am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/4 "2022-02-08T02:15:37Z")

</div>

Can you try removing the mutate+split and replacing it with

```
mutate { gsub => ["message", "^<\d+>", ""] }
json { source => "message" }
```

---

<div class="post-metadata">

**Author:** ![personal.riz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/personal.riz/32/98989_2.png) [@personal.riz](https://discuss.elastic.co/u/personal.riz)\
**Post date:** [February 8, 2022, 5:00am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/5 "2022-02-08T05:00:32Z")

</div>

Thanks a lot Badger, it works.!!! after refresh indexing some fields just appeared

```auto
input {
  tcp {
    port => 5000
  }
}

filter {
  mutate { gsub => ["message", "^<\d+>", ""] }
  json { source => "message" }
}

output {
  elasticsearch{
    hosts => "xxx.xxx.xxx.xxx:9200"
    index => "ngfw"
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2022, 5:00am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/6 "2022-02-08T05:00:32Z")

</div>

Opensearch is an AWS run product and differs from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2022, 5:01am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456/7 "2022-03-08T05:01:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.