# Regex to match a specific key value pair in Logstash

**URL:** <https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167>\
**Category:** Logstash\
**Created:** [May 28, 2019, 6:52pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167 "2019-05-28T18:52:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [May 28, 2019, 6:52pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/1 "2019-05-28T18:52:17Z")

</div>

I'm looking to remove specific key=value pair that are inside a STRING.

Say, input event is as follows:

```
{
	"ABC": "10119707",
	"Request_StartTime": "1558952196175",
	"Severity": "INFO",
	"UUID": "481e8cfa-399c-4996-a4d3-7e9b7ec866fa",
	"Src_LogMsg": "type=abc, user=abc.def, vid=1111, api=fooapi, email=abc.def@gmail.com, cat=1",
	"@version": "1",
	"@timestamp": "2019-05-27T10:16:36.180Z",
	"Src_Host": "Hostname",
	"Request_IpAddress": "1.1.1.1"
}

```

I want to remove the key=value pair `user=abc.def` from the `Src_LogMsg` string. The following works:

```
filter {
  if [Src_LogMsg] =~ /.+/ {
    mutate {
      gsub => ["Src_LogMsg","(user=(.+?)\s)",""]
    }
  }

```

But if the `user=abc.def` is at the `end` of `Src_LogMsg` as opposed to being in `middle`, then the above doesn't work. Please see the below screenshots:

Here `user=abc.def` is in the middle with `cat=1` being the last k=v pair

 ![43%20AM](https://us1.discourse-cdn.com/elastic/original/3X/d/b/db2f261348e3b40001756b42646ee80f094d6204.png)

Here `user=abc.def` is at the end of `Src_LogMsg` string. It's not removed.

 ![05%20AM](https://us1.discourse-cdn.com/elastic/original/3X/6/7/67d177cf8fb21ce14fdf16ec103fffcfab895b5e.png)

Test string from which user=xyz is successfully removed

> {"ABC": "10119707", "Request\_StartTime": "1558952196175", "Severity": "INFO", "UUID": "481e8cfa-399c-4996-a4d3-7e9b7ec866fa", "Src\_LogMsg": "type=abc, user=abc.def, vid=1111, api=fooapi, [email=abc.def@gmail.com](mailto:email=abc.def@gmail.com), cat=1", "@version": "1", "@timestamp": "2019-05-27T10:16:36.180Z", "Src\_Host": "Hostname","Request\_IpAddress": "1.1.1.1"}

Test string from which user=xyz is NOT removed:

> {"ABC": "10119707", "Request\_StartTime": "1558952196175", "Severity": "INFO", "UUID": "481e8cfa-399c-4996-a4d3-7e9b7ec866fa", "Src\_LogMsg": "type=abc, vid=1111, api=fooapi, [email=abc.def@gmail.com](mailto:email=abc.def@gmail.com), cat=1, user=abc.def", "@version": "1", "@timestamp": "2019-05-27T10:16:36.180Z", "Src\_Host": "Hostname","Request\_IpAddress": "1.1.1.1"}

Can someone please help me form the correct regex that will remove the user=abc.def k=v pair irrespective of its location within the `Src_LogMsg` field.

**Logstash.conf:**

```
input {
  stdin {
    codec => json
  }
}

filter {
  if [Src_LogMsg] =~ /.+/ {
    mutate {
      gsub => ["Src_LogMsg","(user=(.+?)\s)",""]
    }
  }
}

output {
  stdout { codec => rubydebug { metadata => true } }
}

```

**Logstash Version: 5.5.1**

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 28, 2019, 9:54pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/2 "2019-05-28T21:54:47Z")

</div>

```
mutate { gsub => ["Src_LogMsg", "user=[^,]+(, |$)", "" ] }
```

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [May 29, 2019, 7:13pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/3 "2019-05-29T19:13:25Z")

</div>

Thank you Badger. Very helpful. However, with this, for the second test case, an extra comma and space appear. Please see below screenshot

**Test Case:**

> {"ABC": "10119707", "Request\_StartTime": "1558952196175", "Severity": "INFO", "UUID": "481e8cfa-399c-4996-a4d3-7e9b7ec866fa", "Src\_LogMsg": "type=abc, vid=1111, api=fooapi, [email=abc.def@gmail.com](mailto:email=abc.def@gmail.com), cat=1, user=abc.def", "@version": "1", "@timestamp": "2019-05-27T10:16:36.180Z", "Src\_Host": "Hostname","Request\_IpAddress": "1.1.1.1"}

 ![17%20PM](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26f6211abd2c020643b6bf8a765c73157c84e4c4.png)

In all, I need to handle 3 cases:

1. `user=abc.def` is at `start` of `Src_LogMsg` string
2. `user=abc.def` is in `middle` of `Src_LogMsg` string
3. `user=abc.def` is at `end` of `Src_LogMsg` string

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 29, 2019, 8:14pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/4 "2019-05-29T20:14:43Z")

</div>

You can use a second regexp to remove the trailing comma and space.

```
mutate { gsub => ["Src_LogMsg", "user=[^,]+(, |$)", "", "Src_LogMsg", ", $", "" ] }

```

Please do not post pictures of text. Just post the text. Thanks!

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [May 29, 2019, 8:39pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/5 "2019-05-29T20:39:40Z")

</div>

It'd be easier to simply overwrite the value with the text `REDACTED` or something, instead of doing multiple passes and accounting for all of the edge-cases.

```auto
filter {
  mutate {
    gsub => ["Src_LogMsg", "(?<=\buser=)[^,]+", "REDACTED"]
  }
}

```

The pattern `(?<=\buser=)[^,]+` literally means "any string of non-comma characters that is immediately proceeded by (a word-break (`\b`) followed by the character sequence `user=`)"

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [May 29, 2019, 8:44pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/6 "2019-05-29T20:44:09Z")

</div>

Excellent suggestion and thanks for the working example. We did think about it at start. But then it means storing dummy fields in ES for Billions of records. Since this is not a field, can't remove it using prune. Thoughts?

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [May 29, 2019, 8:46pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/7 "2019-05-29T20:46:24Z")

</div>

Excellent. Thank you Badger. I also want to remove `email=abc.def@gmail.com` field and so I did the following

```
 gsub => ["Src_LogMsg", "(email=[^,]+(, |$))|(user=[^,]+(, |$))", "", "Src_LogMsg", ", $", "" ]

```

Not sure if this is the most efficient way to do.

And thank you for the note that "text" is better. Agree.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [May 29, 2019, 11:34pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/8 "2019-05-29T23:34:32Z")

</div>

This pattern is a little more sussinct (and formatted to see the multiple phases separately)

```auto
gsub => [
  "Src_LogMsg", "(\b(email|user)=[^,]+(, |$))", "",
  "Src_LogMsg", ", $", ""
]

```

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [May 30, 2019, 12:56pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/9 "2019-05-30T12:56:13Z")

</div>

This is great. Thank you v much! Makes the code very much succinct and easier to read. And more fields can be easily added.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 27, 2019, 12:56pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167/10 "2019-06-27T12:56:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.