To modify multiple field names through logstash

Elastic Stack Logstash
1

Hello,

I came across an old discussion -> Rename or Change Available Fields . Unfortunately in that discussion, the final configuration that worked has not been shared!
I am working on same and wanted to get rid of jsonPayload from the beginning of all fields that starts with it. Basically I just want to remove it.
Per my understanding, by using mutate plugin it doesn't allow to use regex and replace the name of multiple fields.
Also, is there a way to change all upper case field names to lower case.

I am using ES version 7.13.1. Sending logs from google cloud using logstash_pubsub plugin and then passing it through the json filter to parse it and split into more detailed fields.
My current config is very simple

      input {
        google_pubsub {
            project_id => "project-name"
            topic => "topic-name-woith-logs"
            subscription => "topic-subscription-name"
            json_key_file => "/etc/logstash/keys/logstash-sa.json"
        }
      }
      filter {
        json {
          source => "message"
          tag_on_failure => ["_jsonparsefailure"]
          skip_on_invalid_json => true
          remove_field => ["message"]
        }

       # to omit publishing cloudaudit logs to ES
       if "cloudaudit.googleapis.com" in [logName] { drop { } }
      }

      output {
        elasticsearch {
          cloud_id => "${ELASTIC_CLOUD_ID}"
          cloud_auth => "${ELASTIC_CLOUD_AUTH}"
          index => "logs-%{[resource][labels][project_id]}_%{[resource][labels][namespace_name]}-%{+YYYY.MM.dd}"
        }
      }

Can some one please help me ?

2

Hi,

If you know the name of the fields in advance, you can use the lowercase option of the mutate plugin documentation here.

If you don't know the name of the fields in advance, use a ruby script should be the best solution.
A code already exist here. If you want to remove jsonPayload. from the beginning of all fields, edit this line of code event.set(k.downcase, processObject(v)) to this

event.set(k.downcase.gsub(/jsonpayload\./, ''), processObject(v))
event.remove(k)

(the second parameter of gsub is 2 single quotes)

3

Thanks for the suggestion Cad!

I have tried using the script, as below
enabled it

      filter {
        ruby {
          path => "/etc/logstash/filters/script.rb"
        }
      }

Then mounted it using a configmap. I could confirm that all field names are changed to lower case and also jsonPayload from beginning of field names have been removed.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.