@Badger
I want to exclude XML & json key-value if key length is greater than 15 char and value length is greater than 100 char. Is this possible can you please suggest way for it.
This will be very similar to the question you asked about filtering on just the key length
ruby {
code => '
event.to_hash.each { |k, v|
if k.length > 15 and v.length > 100
event.remove(k)
end
}
'
}@Badger ,
It is not working on data which is parsed through XML and JSON filter
That really does not provide enough information for me to suggest a solution. What does the data look like? What do you mean by "not working"?
@Badger ,
I trying to parse below XML message with XML filter:
2021-01-07 05:55:06,931 BLZMUSTANG MONITOR Method=gk.RiskManagement.rdVendorGatewayCaller::Dispose SoapAction=gk.RiskManagement/IBlazeWcfService002/LogonRiskSessionInit PCPNumber=424049880 SessionID=iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0 ChannelVersion=9811 Message=Performance MethodName=ParseCustomFieldsByVendor ResponseTimeInMS=0 Thread=7216 ProcessId=64624 /LM/W3SVC/2/ROOT-1-132542254620449814 WUWVC9ASBLZ01
2021-01-07 05:55:06,931 BLZMUSTANG MONITOR Method=gk.RiskManagement.rdVendorGatewayCaller::CallVendor SoapAction=gk.RiskManagement/IBlazeWcfService002/LogonRiskSessionInit PCPNumber=424049880 SessionID=iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0 ChannelVersion=9811 Message=PERFMON MessageCategory=VendorCalls Source=Mustang Status=Success VIC=Iovation reqTime=2021-01-07 05:55:06.6455223 respTime=2021-01-07 05:55:06.9310857 reqCollPath=VendorCalls,Iovation,Current,Request respCollPath=VendorCalls,Iovation,Current,Response VGRequest=<?xml version="1.0"?>
<requestType xmlns:w="gk.RiskManagement">
<header xmlns="http://www.gk.com/eai/custom/VendorGatewayCDM.xsd">
<applicationID>BLAZE001</applicationID>
<hostname>WUWVC9ASBLZ01</hostname>
<timestamp>2021-01-07T05:55:06.6455223-05:00</timestamp>
<transactionID>8143bd16-d21f-41ae-a30f-084505030d9a</transactionID>
<correlationID>iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0</correlationID>
</header>
<credential xmlns="http://www.gk.com/eai/custom/VendorGatewayCDM.xsd">
<userName>******</userName>
<password>******</password>
</credential>
<service xmlns="http://www.gk.com/eai/custom/VendorGatewayCDM.xsd">GetDeviceFingerPrints</service>
<serviceFunction xmlns="http://www.gk.com/eai/custom/VendorGatewayCDM.xsd">CheckTransactionDetails</serviceFunction>
<provider xmlns="http://www.gk.com/eai/custom/VendorGatewayCDM.xsd">Iovation</provider>
<inputParameter xmlns="http://www.gk.com/eai/custom/VendorGatewayCDM.xsd">
<parameter>
<name>SessionID</name>
<value>iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0</value>
<type>STRINGTYPE</type>
</parameter>
<parameter>
<name>blackboxdata</name>
<value>0660BZA4AsybKO6AwZ3IfUmXUnnh7k2L1Q7wEKRbe1x+WzgANSgC/b5QG/VscFElEKyxRTxRG5VO083k9gXFOvAPwOspt5YyZjrWawBiqhhOisaHhTDX1hvhwEExwcqApi6JYpCkUDK2gK2obeliktMaQOmBUiU6gy5PC28UZZeD18KGQwkDh/NoUOkZBQhPxw57N8UOqGKFSqrH0QunbPb/GShjQ7UxIk+3DATp+z7gr7B3vxuiJYKAtZu+5WH0P38+DuMba1CMtzgEPJ9YvtAgZnZNG3ZcS6v4/NwZhh5C9HP0i98QwROlVO7ecD17RJM52BsqsU1JgcsA3j1Ez+hqJjFm5/ysJxg4N4YBxUZDOD1HA4ksk2ONU2i7XbHwBP6j2E4kK0zco7G9uAaQ2Q763Cc7eGv/KFDMh1RtWmvhHQ2V8zo0gfK4rP/FTzn7/kGDIHEVWqpGkaaNH2aB6nelJbb1mL7gIZNf94x5VjggVFsr4HDAmMlD5twnR9xPX/s0iwuyIY3ezlGnNXxVTyc0i1acdREto4pxAzkdH5BG/3qWcV5bYmENkrEhR5TjVT6OcGiI7KEpxLcGD3BpDpQV7QNh1Afk5Oyw2lAbF/C+BDQWdxUNp1KLTftHIIxRoRMtRJD1hqD/vrW0LE6rmFHZYTlgeV8DXVggyA0ZzfzCX6BjC4RJnA/AT2byU3YXW6jTzZ/mGtOP9sz+CK7MCGLJ87ONT08nTR88f5PrwsK0VMFIQuMD+kAKIVruSkOhJrB5EWP53tGGB9BNbffpRU+qTwUxrkItKqdhsDdfQ2bK37zruaAV4bWWRx2NSfXSInc32rakyvxjTo9H9q4/OztKRIugb2I4svJtJZ0b1kLAQqP3LGHVSzVMUVfB5mOVhViZV2uLjtqF+AI1fIk3LgGH/z9o1S6vhmYieBve0fahI7xAduXEIRQ040AFt7+cZZX2N6OMjQnt53BJWT4TyheVg2KIoq7d43JxHOh7ZMvyxsrdl38SCFT55F4Vd1K2OYXr6V+oHxwvhAIX7Zl91dTDiyK4GrGaCzN0RQTghoNfAyVzhcCELLjeRPcq9GD95++f+rzPlQVj4Zn6xp5Kupzi1SpUM33zJ5qb6L6EfaD0kACPIM9SuUgHj0sK+5JquClefL4zJBJ8cNGtbc7zJkYjkwgwhWUkVha5x1jJbGcsIGbDL4Cq+nd4EGw+gEXJGNA97nzC3mXMSQXaheZUWlTvzVd6uPSgggRTP1uPyxRHhY1Pr37gep+ery5oVhOnHnOmObBAN8uQ4ewBIu2/Uz/qAgx4qP9287NnEiOH8bf3Dvc6kZ7b/rlBHQU5M9aOdC3RSngJho6KPTQhBIYBYE8UY/NYMUXvctZ+NP5YFj2HUEJPdXBLU/ZtdobUZmuFV7volCMKYgchR90fQflnXADAc7GpltmV3R1CgNm7EPkjTTJkFyo2P7zRbCFLXRHaG87aMF1M2mFpcE2hYFazCs3A</value>
<type>STRINGTYPE</type>
</parameter>
<parameter>
<name>customer_number</name>
<value>424049880</value>
<type>STRINGTYPE</type>
</parameter>
<parameter>
<name>enduser_ip</name>
<value>158.14*****.232</value>
<type>STRINGTYPE</type>
</parameter>
<parameter>
<name>call_type</name>
<value>transfer</value>
<type>STRINGTYPE</type>
</parameter>
<parameter>
<name>PCPNumber</name>
<value>424049880</value>
<type>STRINGTYPE</type>
</parameter>
</inputParameter>
</requestType>
Grok Pattern:
if "VGRequest" in [message]{
grok{
match => { "message" => "(?m)%{TIMESTAMP_ISO8601:appTimestamp}\s*%{WORD:AppName}\s*%{WORD:Loglevel}\s%{GREEDYDATA:logmessage}\s*VGRequest=%{GREEDYDATA:xmlmessage}"}
}
xml {
source => "xmlmessage"
target => "xml"
}
}
Here I want to exclude fields (Parsed by xml filter) if char are greater than 15 and fields value greater than 100
Please edit your post, select the "XML" and click on </> in the tool bar above the edit pane. If you look at your post you will see that there are no XML tags at the moment.
@Badger ,
Can you please check it now
OK, so you want to recursively remove long keys/values from your event. That will require a ruby function. That is going to be similar to this.
xml { source => "message" target => "xml" force_array => false }
ruby {
code => '
def removeBigThings(object, name, event)
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
removeBigThings(object[i], "#{name}[#{i}]", event)
}
else
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length > 15 or object.to_s.length > 100
event.remove(name)
end
end
end
end
event.to_hash.each { |k, v|
removeBigThings(v, "[#{k}]", event)
}
'
}
Note that I have used force_array => false. You do not have to, I just do not like everything being an array with one member.
Note also that I used "or" in the test lastElement.length > 15 or object.to_s.length > 100. You said "and" but that would result in nothing being removed.
Note also that this will delete [message], since that is more than 100 characters long. If you only want to modify things in [xml] you could change
event.to_hash.each { |k, v|
removeBigThings(v, "[#{k}]", event)
}
to
removeBigThings(event.get("xml"), "[xml]", event)
1 Like
@Badger ,
I used above code as it is but is not dropping keys if greater than 15 . Worked for values
If, after
def removeBigThings(object, name, event)
you add a line
puts "Called for #{name}"
do you see the function being called for each object in the event?
@Badger ,
I am using following code but it is not drooping keys which are greater than 15.
fields created:
Sample filed: xml.outputParameter.vendorResponse.provider.keyword
ruby {
code => '
def removeBigThings(object, name, event)
puts "Called for #{name}"
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
removeBigThings(object[i], "#{name}[#{i}]", event)
}
else
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length > 15 or object.to_s.length > 30
event.remove(name)
end
end
end
end
removeBigThings(event.get("xml"), "[xml]", event)
'
}So if you have an [a2345678] field inside a [b2345678] object, for a total of 17 characters when represented as a2345678.b2345678 you want to drop it?
If so you will need to change
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length > 15
to something more like
dottedName = name.gsub(/\]\[/, ".").gsub(/\]$/, "").gsub(/^\[/, "")
if dottedName.length > 15@Badger ,
Yes above understanding is correct
I have added following script but still it is not providing
desired result.
ruby {
code => '
def removeBigThings(object, name, event)
puts "Called for #{name}"
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
removeBigThings(object[i], "#{name}[#{i}]", event)
}
else
dottedName = name.gsub(/\]\[/, ".").gsub(/\]$/, "").gsub(/^\[/, "")
if dottedName.length > 15 or object.to_s.length > 30
event.remove(name)
end
end
end
end
removeBigThings(event.get("xml"), "[xml]", event)
'
}I do not know what to say. If I add a line
puts "Removing #{dottedName}"
immediately before the event.remove then I can see it removes everything that meets the condition
Called for [xml]
Called for [xml][service]
Called for [xml][service][xmlns]
Removing xml.service.xmlns
Called for [xml][service][content]
Removing xml.service.content
Called for [xml][xmlns:w]
Called for [xml][inputParameter]
Called for [xml][inputParameter][xmlns]
Removing xml.inputParameter.xmlns
etc. That leaves
"xml" => {
"credential" => {},
"inputParameter" => {
"parameter" => [
[0] {},
[1] {},
[2] {},
[3] {},
[4] {},
[5] {}
]
},
"provider" => {},
"service" => {},
"serviceFunction" => {},
"xmlns:w" => "gk.RiskManagement",
"header" => {}
},@Badger ,
Really appreciate your quick help every time. it is working as expected.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.