How add prefix to the fields using kv filter

Elastic Stack Logstash
1

Hi
I am adding prefix to the fields using kv filter, It's working to the some of the fields in the json message.
Example: Filed name is resource.network.imageid while adding prefix as aws.datasource using kv filter I got the output as aws.datasource.imageid but I want output as aws.datasource.resource.network.imageid.
In filter section I am using below kv filter

    kv {
     source => "message"
    field_split_pattern => ".,"
      value_split => ":"
       prefix => "aws.datasource"
       remove_char_key => "\"{"
       remove_char_value => "\[{}[],\"]"
       trim_key => "\"{"
       trim_value => "\[{}[\*],\"]"
       }
2

Please edit your post, select the filter configuration, and click on </> in the toolbar. That will result in the configuration looking like

kv {
    source => "message"

etc.

Can you show us what a message looks like in

output { stdout { codec => rubydebug } }
3

Hey Badger,
I am using the below configuration file.

  input {
        file {
            path => "/root/logs.txt"
             start_position => "beginning"
             sincedb_path => "/tmp/csv"
      }

      }
    }
    filter {

          json {
            source => "message"

           }

    kv {
     source => "message"
    field_split_pattern => ".,\[^resource]"
      value_split => ":"
       prefix => "aws.datasource."
       remove_char_key => "\"{"
       remove_char_value => "\[{}[],\"]"
       trim_key => "\"{"
       trim_value => "\[{}[],\"]"
       }
    }
    output {
          elasticsearch {
                hosts => ["xxxxxxxx:9200"]
                index => "test-"
                user => "xxxx"
                password => "xxxx"
            }
    stdout { codec => rubydebug }
    }

{"schemaVersion":"2.0","accountId":"345612345","region":"northeast-1","partition":"aws","id":"20456vghu90sders3a0b14a54f74d47","arn":"arn:aws:datasource:northeast-1:997405263248:detector/58b78d709b32f0224388a7bf82d44843/finding/20b8990ff15f880d93a0b14a54f74d47","type":"UnauthorizedAccess:IAMUser/ConsoleLogin","resource":{"resourceType":"AccessKey","accessKeyDetails":{"accessKeyId":null,"principalId":"ARABRWNYSFGFSCGGGKCZTBH6:logstash@team.com","userType":"AssumedRole","userName":"logstash@team.com"}},"service":{"serviceName":"guardduty","detectorId":"58b78d709ub50scd23582d44843","action":{"actionType":"AWS_API_CALL","awsApiCallAction":{"api":"ConsoleLogin","serviceName":"signin.amazonaws.com","callerType":"Remote IP","remoteIpDetails":{"ipAddressV4":"17.71.13.157","organization":{"asn":"0022","asnOrg":"COMCAST-7922","isp":"Comcast Cable"

The output Iam expecting
aws.datasource.service.action.actionType:AWS_API_CALL
aws.datasource.resource.resourcetype:Accesskey

4

I don't think you can do that with a kv filter. However, the json filter will parse the JSON and you can then move the fields around. Either

    mutate {
        copy => {
            "[service][action][actionType]" => "aws.datasource.service.action.actionType"
            "[resource][resourceType]" => "aws.datasource.resource.resourcetype"
        }
    }

if you want field names with dots in them

    mutate {
        copy => {
            "[service][action][actionType]" => "[aws][datasource][service][action][actionType]"
            "[resource][resourceType]" => "[aws][datasource][resource][resourcetype]"
        }
    }

if you want fields nested in an object.

5

You are saying that we can't use kv filter for nexted field in the data,you saying need to define aws.datasource manually but I want to add dynamically because I am getting so may fields. Is there any possibility to add dynamically.

6

You could use a ruby filter to iterate over the fields and rename them.

7

ok,I have tried with ruby code but I didn't get any output.Can you please provide code related to the adding prefix to the field.

8

you can try target => "[aws][datasource]" inside your kv filter