How to remove fields with regex from json?

Elastic Stack Logstash
1

Hi,

I am stucking in removing fields with regex. I have log in json. From filebeat i have configured it to push to logstash via config:

filebeat.prospectors:

  • input_type: log
    paths:
    • E:\test\app\Logging\test.json
      json.keys_under_root: true
      json.overwrite_keys: false
      json.add_error_key: true

The log content will be something like this:

{"hi":"hello","test":{"qs: abc":"1","qs:_12313":"2", "demo":"test"}}

nested object "qs" is random changed and I want to remove them, just keep "demo" and "hi".

Could you advise me how to remove it please?

Thanks in advance.

Son

2

Can somebody help me please?

I am trying with below ruby config but i don't know how to remove subfield due to the eventapi has changed from version 5.x:

ruby {
code => "
if event.get('test') != nil
event.get('test').to_hash.keys.each { |k|
if k =~ /qs*/
event.remove(k)
end
}
end
"
}
}

Thanks in advance.

3

Do you try with mutate plugin?

muate {
   remove_field => ["test.qs"]   
}
4

Hi,

"qs" is just prefix, there are random strings trailing (qs: abc, qs:_demo, qs: 123 and more). That's why i don't know how to remove with regex. I gave it some tries with other plugin but no luck.

Could you advise please?

Thanks in advance.

5

Ive never tried this on nested objects, but when I ingest log files I filter out noise(pings, etc) by using

        if [message] =~ "blah"  {
                    grok {
            match => { "message" => "%{BASE10NUM:Stored_Procedure_Exec_Time_Ms}" }
        }
    }

I used the grok filter here, but you should look into the drop filter. Just as a reminder, when you deal with nested objects in logstash, you should use [field][nestedField] as the format.

6

Hi Cody,

There is no regex for nested object as i checked. That's why i had to use ruby filter. But unfortunately i don't know ruby so it is my issue now. I cannot remove nested field with ruby code.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.