Using wildcard to update/remove fields inside logstash filter

Hi
I have a data structure like this. This is an stdout after parsing with XML filter.

{
"offset" => 6859148,
"input_type" => "log",
"source" => "requests.log.6",
"type" => "log",
"datestamp" => "08-09-17 03:03:07.401",
"@timestamp" => 2017-10-17T18:32:04.491Z,
"@version" => "1",
"msg_type" => "REQUEST",
"parsed_msg" => {
"Body" => [
[0] {
"getIndContents" => [
[0] {
"returnXML" => [
[0] "true"
],
"indexName" => [
[0] "Some Index"
],
"fieldNames" => [
[0] {
"JavaLangstring" => [
[ 0] "Out$float",
[ 1] "Mark$float",
[ 2] "Pct$float",
[ 3] "Description$String",
[ 4] "Is$String",
[ 5] "Preferred_Id$String",
[ 6] "Co$float",
[ 7] "Ma$date",
[ 8] "Pr$float",
[ 9] "Yi$float",
[10] "OA$float",
[11] "Dur$float",
[12] "Mo$String@CODE",
[13] "SP$String@CODE",
[14] "Mar$float",
[15] "Pct$float"
]
}
],
"login" => [
[0] "index_website"
],
"userName" => [
[0] "System"
],
"version" => [
[0] "1.1"
],
"xmlns" => "http://somewebsite.com",
"site" => [
[0] "abcsite"
],
"universe" => [
[0] "1"
],
"currency" => [
[0] "USD"
],
"pwd" => [
[0] "pass123"
],
"asOfDate" => [
[0] "07/31/2017"
]
}
]
}
]
},
"request_id" => "7771f2ca-7e45-4479-b7c7-91566b0f89ee"
},
{
"offset" => 1275245,
"input_type" => "log",
"source" => "requests.log.8",
"type" => "log",
"datestamp" => "08-08-17 17:58:17.405",
"@timestamp" => 2017-10-17T18:32:01.769Z,
"@version" => "1",
"msg_type" => "REQUEST",
"parsed_msg" => {
"Body" => [
[0] {
"getBiginfo" => [
[0] {
"xmlns" => "somewebsite.com",
"site" => [
[0] "sometest"
],
"ownerOnly" => [
[0] "true"
],
"login" => [
[0] "in_web"
],
"pwd" => [
[0] "Pass123"
],
"userName" => [
[0] "CHECK1"
],
"version" => [
[0] "1.1"
],
"typeCode" => [
[0] "LL_DEFINITION"
]
}
]
}
]
},
{
"offset" => 12233404,
"input_type" => "log",
"source" => "requests.log.8",
"type" => "log",
"datestamp" => "08-08-17 00:21:43.031",
"@timestamp" => 2017-10-17T18:32:04.528Z,
"@version" => "1",
"msg_type" => "REQUEST",
"parsed_msg" => {
"Body" => [
[0] {
"getAllForUsr" => [
[0] {
"xmlns" => "somewebsite.com",
"site" => [
[0] {
"xsi:nil" => "true"
}
],
"login" => [
[0] "in_web"
],
"pwd" => [
[0] "BbBbBbB"
],
"userName" => [
[0] "UMEMA"
],
"version" => [
[0] "1.1"
]
}
]
}
]
},
"request_id" => "c62a2214-e111-48a9-b9f9-0e28791eb7e2"
}

I am able to do this :
mutate{
update => { "[parsed_msg][Body][0]" => "XXXXXXXXX" }
}

which replaces the contents of entire Body[0] with "XXXXXXX"
But the further inside Body[0] here "getIndContents" or "GetAllForUsr" etc is a method name which can vary based on the method being called by the application so i wanna use a wildcard to traverse down to pwd field and mask it out which doesn't seem to work.
I tried all possible combinations like Body[0][*][0][pwd][0] but later found out that wildcards are not supported.
How can I either remove the pwd field or update it to have XXXXXX ?

Wildcards aren't supported. You need to use a ruby filter to traverse the datastructures with code. Examples of this have been posted in the past.

I tried to look around and found out what i'm trying to achieve is possible through a recursive function.
However i cannot find any examples on ruby functions inside a logstash config.
Would you be able to point me to an example where it is possible to reach a node for example ("pwd") where the path to it can vary across events ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.