# How to remove data string if NULL from user-agent filtered

**URL:** <https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400>\
**Category:** Logstash\
**Created:** [September 13, 2018, 3:23am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400 "2018-09-13T03:23:49Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [September 13, 2018, 3:23am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/1 "2018-09-13T03:23:49Z")

</div>

Hi Guys,

Is there way to remove, OS field **%{os\_major}.%{os\_minor}** and NAME field **%{major}.%{minor}** of user-agent?

if the value is NULL? Because, it looks like this on my Kibana, kinda irritating on my eyes.

 ![filter_data](https://us1.discourse-cdn.com/elastic/original/3X/3/9/39fa228990feadaf8f871591afee1940b6d5d80c.png)

Below are the syntax, how I've filtered the data from logstash.

mutate {  
add\_field =\> { "browser\_name.version" =\> "%{name} %{major}.%{minor}" }  
add\_field =\> { "os\_name.version" =\> "%{os} %{os\_major}.%{os\_minor}" }  
}

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 14, 2018, 5:00am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/2 "2018-09-14T05:00:26Z")

</div>

Because the values at the addresses provided in the sprintf format string do not exist, they are not getting replaced.

One way to get around this is to only perform the mutate if the fields you need are present:

```auto
filter {
  if [os] and [os_major] and [os_minor] {
    mutate {
      add_field {
        "os_name.version" => "%{os} %{os_major}.%{os_minor}"
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [September 14, 2018, 6:32am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/3 "2018-09-14T06:32:48Z")

</div>

Hi @yaauie ,

Thanks for the information.

Can you help me again please, I have another issue, I can't filter the highlighted **username** using below pipeline.conf.

**username** located in the request\_field.

Example:

/api/v2/members/ **y23228020** /wallets/0/balance  
/api/v2/members/ **py51171** /wallets

Or is it possible to filter username thru **scripted\_field**? and how?

 ![](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6e3200b834e7aff5dd8c4171f690a2892a0d20f2.png)

input {

beats {

port =\> "5044"

}

}

filter {

grok {

match =\> { "message" =\> "%{COMBINEDAPACHELOG}"}

}

geoip {

source =\> "clientip"

}

useragent {

source =\> "agent"

}

}

output {

elasticsearch {

hosts =\> ["localhost:9200"]

manage\_template =\> false

index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

}

}

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 14, 2018, 5:04pm UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/4 "2018-09-14T17:04:11Z")

</div>

Please open a new thread with your separate issue; that way your new question will have a wider audience.

When you do, it would be helpful to include what you have tried so far, how it is behaving differently than you expect, and the event's current output shape (e.g., using `output { stdout { codec => rubydebug }}`).

* * *

To get you started, my best guess is that your `COMINEDAPACHELOG` grok is providing a field `request`, which contains the relative request URI. Since you are looking to extract information from this URI, you will likely have a _second_ grok filter to perform this action, likely guarded by an `if` clause to ensure the plugin doesn't run on events that don't match:

```auto
filter {
   # ...
  if [request] and [request] =~ /members/
    grok {
      match {
        "request" => "^/api/v2/members/%{WORD:request_member_id}/"
      }
    }
  }

```

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [September 18, 2018, 6:16am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/5 "2018-09-18T06:16:29Z")

</div>

Hi @yaauie,

Thanks for your input. I tried that to put on my pipe.conf.

But, what I've encountered was, no data had been filtered since there were no any messages displayed on my kibana once I restarted my logstash. Config was OK no any error found.

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [September 19, 2018, 6:56am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/6 "2018-09-19T06:56:42Z")

</div>

@yaauie,

Thanks for your idea, all good now. I just removed the If condition and everything goes to my expected output.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 19, 2018, 11:37pm UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/7 "2018-09-19T23:37:49Z")

</div>

it looks like the if condition needs parentheses to group the clauses:

```auto
if [request] and ([request] =~ /members/)
    grok {
      match {
        "request" => "^/api/v2/members/%{WORD:request_member_id}/"
      }
    }
  }

```

the if clause is helpful here, because if you encounter messages that do _not_ have a request field with `members` in it, they'll (a) expend extra, unnecessary work and (b) tag each event with `_grokparsefailure`, which may not be helpful.

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [September 21, 2018, 2:34am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/8 "2018-09-21T02:34:15Z")

</div>

@yaauie,

That's noted. Thanks for the info.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2018, 2:34am UTC](https://discuss.elastic.co/t/how-to-remove-data-string-if-null-from-user-agent-filtered/148400/9 "2018-10-19T02:34:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.