Hi
I am adding prefix to the fields using kv filter, It's working to the some of the fields in the json message.
Example: Filed name is resource.network.imageid while adding prefix as aws.datasource using kv filter I got the output as aws.datasource.imageid but I want output as aws.datasource.resource.network.imageid.
In filter section I am using below kv filter
kv {
source => "message"
field_split_pattern => ".,"
value_split => ":"
prefix => "aws.datasource"
remove_char_key => "\"{"
remove_char_value => "\[{}[],\"]"
trim_key => "\"{"
trim_value => "\[{}[\*],\"]"
}
Badger
June 22, 2020, 8:40pm
2
Please edit your post, select the filter configuration, and click on </> in the toolbar. That will result in the configuration looking like
kv {
source => "message"
etc.
Can you show us what a message looks like in
output { stdout { codec => rubydebug } }
Hey Badger,
I am using the below configuration file.
input {
file {
path => "/root/logs.txt"
start_position => "beginning"
sincedb_path => "/tmp/csv"
}
}
}
filter {
json {
source => "message"
}
kv {
source => "message"
field_split_pattern => ".,\[^resource]"
value_split => ":"
prefix => "aws.datasource."
remove_char_key => "\"{"
remove_char_value => "\[{}[],\"]"
trim_key => "\"{"
trim_value => "\[{}[],\"]"
}
}
output {
elasticsearch {
hosts => ["xxxxxxxx:9200"]
index => "test-"
user => "xxxx"
password => "xxxx"
}
stdout { codec => rubydebug }
}
{"schemaVersion":"2.0","accountId":"345612345","region":"northeast-1","partition":"aws","id":"20456vghu90sders3a0b14a54f74d47","arn":"arn:aws:datasource:northeast-1:997405263248:detector/58b78d709b32f0224388a7bf82d44843/finding/20b8990ff15f880d93a0b14a54f74d47","type":"UnauthorizedAccess:IAMUser/ConsoleLogin","resource":{"resourceType":"AccessKey","accessKeyDetails":{"accessKeyId":null,"principalId":"ARABRWNYSFGFSCGGGKCZTBH6:logstash@team.com ","userType":"AssumedRole","userName":"logstash@team.com "}},"service":{"serviceName":"guardduty","detectorId":"58b78d709ub50scd23582d44843","action":{"actionType":"AWS_API_CALL","awsApiCallAction":{"api":"ConsoleLogin","serviceName":"signin.amazonaws.com ","callerType":"Remote IP","remoteIpDetails":{"ipAddressV4":"17.71.13.157","organization":{"asn":"0022","asnOrg":"COMCAST-7922","isp":"Comcast Cable"
The output Iam expecting
aws.datasource.service.action.actionType:AWS_API_CALL
aws.datasource.resource.resourcetype:Accesskey
Badger
June 22, 2020, 10:03pm
4
I don't think you can do that with a kv filter. However, the json filter will parse the JSON and you can then move the fields around. Either
mutate {
copy => {
"[service][action][actionType]" => "aws.datasource.service.action.actionType"
"[resource][resourceType]" => "aws.datasource.resource.resourcetype"
}
}
if you want field names with dots in them
mutate {
copy => {
"[service][action][actionType]" => "[aws][datasource][service][action][actionType]"
"[resource][resourceType]" => "[aws][datasource][resource][resourcetype]"
}
}
if you want fields nested in an object.
You are saying that we can't use kv filter for nexted field in the data,you saying need to define aws.datasource manually but I want to add dynamically because I am getting so may fields. Is there any possibility to add dynamically.
Badger
June 23, 2020, 1:37pm
6
You could use a ruby filter to iterate over the fields and rename them.
ok,I have tried with ruby code but I didn't get any output.Can you please provide code related to the adding prefix to the field.
you can try target => "[aws][datasource]"
inside your kv filter