How to remove data string if NULL from user-agent filtered

[mark.quilates (Mr.M)]

Hi Guys,

Is there way to remove, OS field %{os_major}.%{os_minor} and NAME field %{major}.%{minor} of user-agent?

if the value is NULL? Because, it looks like this on my Kibana, kinda irritating on my eyes.

Below are the syntax, how I've filtered the data from logstash.

mutate {
add_field => { "browser_name.version" => "%{name} %{major}.%{minor}" }
add_field => { "os_name.version" => "%{os} %{os_major}.%{os_minor}" }
}

[yaauie (Ry Biesemeyer)]

Because the values at the addresses provided in the sprintf format string do not exist, they are not getting replaced.

One way to get around this is to only perform the mutate if the fields you need are present:

filter {
  if [os] and [os_major] and [os_minor] {
    mutate {
      add_field {
        "os_name.version" => "%{os} %{os_major}.%{os_minor}"
      }
    }
  }
}

[mark.quilates (Mr.M)]

Hi @yaauie ,

Thanks for the information.

Can you help me again please, I have another issue, I can't filter the highlighted username using below pipeline.conf.

username located in the request_field.

Example:

/api/v2/members/y23228020/wallets/0/balance
/api/v2/members/py51171/wallets

Or is it possible to filter username thru scripted_field? and how?

input {

beats {

port => "5044"

}

}

filter {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}"}

}

geoip {

source => "clientip"

}

useragent {

source => "agent"

}

}

output {

elasticsearch {

hosts => [ "localhost:9200" ]

manage_template => false

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

}

}

[yaauie (Ry Biesemeyer)]

Please open a new thread with your separate issue; that way your new question will have a wider audience.

When you do, it would be helpful to include what you have tried so far, how it is behaving differently than you expect, and the event's current output shape (e.g., using output { stdout { codec => rubydebug }}).

---

To get you started, my best guess is that your COMINEDAPACHELOG grok is providing a field request, which contains the relative request URI. Since you are looking to extract information from this URI, you will likely have a second grok filter to perform this action, likely guarded by an if clause to ensure the plugin doesn't run on events that don't match:

filter {
   # ...
  if [request] and [request] =~ /members/
    grok {
      match {
        "request" => "^/api/v2/members/%{WORD:request_member_id}/"
      }
    }
  }

[mark.quilates (Mr.M)]

Hi @yaauie,

Thanks for your input. I tried that to put on my pipe.conf.

But, what I've encountered was, no data had been filtered since there were no any messages displayed on my kibana once I restarted my logstash. Config was OK no any error found.

[mark.quilates (Mr.M)]

@yaauie,

Thanks for your idea, all good now. I just removed the If condition and everything goes to my expected output.

[yaauie (Ry Biesemeyer)]

it looks like the if condition needs parentheses to group the clauses:

if [request] and ([request] =~ /members/)
    grok {
      match {
        "request" => "^/api/v2/members/%{WORD:request_member_id}/"
      }
    }
  }

the if clause is helpful here, because if you encounter messages that do not have a request field with members in it, they'll (a) expend extra, unnecessary work and (b) tag each event with _grokparsefailure, which may not be helpful.

[mark.quilates (Mr.M)]

@yaauie,

That's noted. Thanks for the info.