# How to exclude XML & json key-value if key length is greater than 15 char and value length is greater than 100 char
**URL:** https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248
**Category:** Logstash
**Created:** [April 15, 2021, 2:11pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248 "2021-04-15T14:11:17Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 15, 2021, 2:11pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/1 "2021-04-15T14:11:17Z")
@Badger
I want to exclude XML & json key-value if key length is greater than 15 char and value length is greater than 100 char. Is this possible can you please suggest way for it.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 15, 2021, 2:57pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/2 "2021-04-15T14:57:23Z")
This will be very similar to the [question](https://discuss.elastic.co/t/how-to-exclude-key-value-pair-from-kv-filter-if-key-size-greater-than-15-char/269566/2) you asked about filtering on just the key length
```
ruby {
code => '
event.to_hash.each { |k, v|
if k.length > 15 and v.length > 100
event.remove(k)
end
}
'
}
```
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 15, 2021, 3:04pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/3 "2021-04-15T15:04:50Z")
@Badger ,
It is not working on data which is parsed through XML and JSON filter
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 15, 2021, 3:09pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/4 "2021-04-15T15:09:09Z")
That really does not provide enough information for me to suggest a solution. What does the data look like? What do you mean by "not working"?
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 15, 2021, 3:37pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/5 "2021-04-15T15:37:51Z")
@Badger ,
I trying to parse below XML message with XML filter:
```
2021-01-07 05:55:06,931 BLZMUSTANG MONITOR Method=gk.RiskManagement.rdVendorGatewayCaller::Dispose SoapAction=gk.RiskManagement/IBlazeWcfService002/LogonRiskSessionInit PCPNumber=424049880 SessionID=iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0 ChannelVersion=9811 Message=Performance MethodName=ParseCustomFieldsByVendor ResponseTimeInMS=0 Thread=7216 ProcessId=64624 /LM/W3SVC/2/ROOT-1-132542254620449814 WUWVC9ASBLZ01
2021-01-07 05:55:06,931 BLZMUSTANG MONITOR Method=gk.RiskManagement.rdVendorGatewayCaller::CallVendor SoapAction=gk.RiskManagement/IBlazeWcfService002/LogonRiskSessionInit PCPNumber=424049880 SessionID=iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0 ChannelVersion=9811 Message=PERFMON MessageCategory=VendorCalls Source=Mustang Status=Success VIC=Iovation reqTime=2021-01-07 05:55:06.6455223 respTime=2021-01-07 05:55:06.9310857 reqCollPath=VendorCalls,Iovation,Current,Request respCollPath=VendorCalls,Iovation,Current,Response VGRequest=
BLAZE001
WUWVC9ASBLZ01
2021-01-07T05:55:06.6455223-05:00
8143bd16-d21f-41ae-a30f-084505030d9a
iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0
******
******
GetDeviceFingerPrints
CheckTransactionDetails
Iovation
SessionID
iph-5a86f20c-3f68-4eeb-ab8a-cf9eb6f639c0
STRINGTYPE
blackboxdata
0660BZA4AsybKO6AwZ3IfUmXUnnh7k2L1Q7wEKRbe1x+WzgANSgC/b5QG/VscFElEKyxRTxRG5VO083k9gXFOvAPwOspt5YyZjrWawBiqhhOisaHhTDX1hvhwEExwcqApi6JYpCkUDK2gK2obeliktMaQOmBUiU6gy5PC28UZZeD18KGQwkDh/NoUOkZBQhPxw57N8UOqGKFSqrH0QunbPb/GShjQ7UxIk+3DATp+z7gr7B3vxuiJYKAtZu+5WH0P38+DuMba1CMtzgEPJ9YvtAgZnZNG3ZcS6v4/NwZhh5C9HP0i98QwROlVO7ecD17RJM52BsqsU1JgcsA3j1Ez+hqJjFm5/ysJxg4N4YBxUZDOD1HA4ksk2ONU2i7XbHwBP6j2E4kK0zco7G9uAaQ2Q763Cc7eGv/KFDMh1RtWmvhHQ2V8zo0gfK4rP/FTzn7/kGDIHEVWqpGkaaNH2aB6nelJbb1mL7gIZNf94x5VjggVFsr4HDAmMlD5twnR9xPX/s0iwuyIY3ezlGnNXxVTyc0i1acdREto4pxAzkdH5BG/3qWcV5bYmENkrEhR5TjVT6OcGiI7KEpxLcGD3BpDpQV7QNh1Afk5Oyw2lAbF/C+BDQWdxUNp1KLTftHIIxRoRMtRJD1hqD/vrW0LE6rmFHZYTlgeV8DXVggyA0ZzfzCX6BjC4RJnA/AT2byU3YXW6jTzZ/mGtOP9sz+CK7MCGLJ87ONT08nTR88f5PrwsK0VMFIQuMD+kAKIVruSkOhJrB5EWP53tGGB9BNbffpRU+qTwUxrkItKqdhsDdfQ2bK37zruaAV4bWWRx2NSfXSInc32rakyvxjTo9H9q4/OztKRIugb2I4svJtJZ0b1kLAQqP3LGHVSzVMUVfB5mOVhViZV2uLjtqF+AI1fIk3LgGH/z9o1S6vhmYieBve0fahI7xAduXEIRQ040AFt7+cZZX2N6OMjQnt53BJWT4TyheVg2KIoq7d43JxHOh7ZMvyxsrdl38SCFT55F4Vd1K2OYXr6V+oHxwvhAIX7Zl91dTDiyK4GrGaCzN0RQTghoNfAyVzhcCELLjeRPcq9GD95++f+rzPlQVj4Zn6xp5Kupzi1SpUM33zJ5qb6L6EfaD0kACPIM9SuUgHj0sK+5JquClefL4zJBJ8cNGtbc7zJkYjkwgwhWUkVha5x1jJbGcsIGbDL4Cq+nd4EGw+gEXJGNA97nzC3mXMSQXaheZUWlTvzVd6uPSgggRTP1uPyxRHhY1Pr37gep+ery5oVhOnHnOmObBAN8uQ4ewBIu2/Uz/qAgx4qP9287NnEiOH8bf3Dvc6kZ7b/rlBHQU5M9aOdC3RSngJho6KPTQhBIYBYE8UY/NYMUXvctZ+NP5YFj2HUEJPdXBLU/ZtdobUZmuFV7volCMKYgchR90fQflnXADAc7GpltmV3R1CgNm7EPkjTTJkFyo2P7zRbCFLXRHaG87aMF1M2mFpcE2hYFazCs3A
STRINGTYPE
customer_number
424049880
STRINGTYPE
enduser_ip
158.14 *****.232
STRINGTYPE
call_type
transfer
STRINGTYPE
PCPNumber
424049880
STRINGTYPE
```
Grok Pattern:
```
if "VGRequest" in [message]{
grok{
match => { "message" => "(?m)%{TIMESTAMP_ISO8601:appTimestamp}\s*%{WORD:AppName}\s*%{WORD:Loglevel}\s%{GREEDYDATA:logmessage}\s*VGRequest=%{GREEDYDATA:xmlmessage}"}
}
xml {
source => "xmlmessage"
target => "xml"
}
}
```
Here I want to exclude fields (Parsed by xml filter) if char are greater than 15 and fields value greater than 100
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 15, 2021, 3:44pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/6 "2021-04-15T15:44:39Z")
Please edit your post, select the "XML" and click on \ in the tool bar above the edit pane. If you look at your post you will see that there are no XML tags at the moment.
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 15, 2021, 4:00pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/7 "2021-04-15T16:00:04Z")
@Badger ,
Can you please check it now
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 15, 2021, 4:44pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/8 "2021-04-15T16:44:15Z")
OK, so you want to recursively remove long keys/values from your event. That will require a ruby function. That is going to be similar to [this](https://discuss.elastic.co/t/to-exclude-around-350-fields-in-json-in-logstash/239019/6).
```
xml { source => "message" target => "xml" force_array => false }
ruby {
code => '
def removeBigThings(object, name, event)
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
removeBigThings(object[i], "#{name}[#{i}]", event)
}
else
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length > 15 or object.to_s.length > 100
event.remove(name)
end
end
end
end
event.to_hash.each { |k, v|
removeBigThings(v, "[#{k}]", event)
}
'
}
```
Note that I have used force\_array =\> false. You do not have to, I just do not like everything being an array with one member.
Note also that I used "or" in the test `lastElement.length > 15 or object.to_s.length > 100`. You said "and" but that would result in nothing being removed.
Note also that this will delete [message], since that is more than 100 characters long. If you only want to modify things in [xml] you could change
```
event.to_hash.each { |k, v|
removeBigThings(v, "[#{k}]", event)
}
```
to
```
removeBigThings(event.get("xml"), "[xml]", event)
```
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 15, 2021, 5:08pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/9 "2021-04-15T17:08:18Z")
@Badger ,
I used above code as it is but is not dropping keys if greater than 15 . Worked for values
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 15, 2021, 5:21pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/10 "2021-04-15T17:21:06Z")
If, after
```
def removeBigThings(object, name, event)
```
you add a line
```
puts "Called for #{name}"
```
do you see the function being called for each object in the event?
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 16, 2021, 5:51am UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/11 "2021-04-16T05:51:45Z")
@Badger ,
I am using following code but it is not drooping keys which are greater than 15.
fields created:
Sample filed: xml.outputParameter.vendorResponse.provider.keyword
```
ruby {
code => '
def removeBigThings(object, name, event)
puts "Called for #{name}"
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
removeBigThings(object[i], "#{name}[#{i}]", event)
}
else
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length > 15 or object.to_s.length > 30
event.remove(name)
end
end
end
end
removeBigThings(event.get("xml"), "[xml]", event)
'
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 16, 2021, 2:39pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/12 "2021-04-16T14:39:31Z")
So if you have an [a2345678] field inside a [b2345678] object, for a total of 17 characters when represented as a2345678.b2345678 you want to drop it?
If so you will need to change
```
lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
if lastElement.length > 15
```
to something more like
```
dottedName = name.gsub(/\]\[/, ".").gsub(/\]$/, "").gsub(/^\[/, "")
if dottedName.length > 15
```
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 16, 2021, 3:01pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/13 "2021-04-16T15:01:54Z")
@Badger ,
Yes above understanding is correct
I have added following script but still it is not providing
desired result.
```
ruby {
code => '
def removeBigThings(object, name, event)
puts "Called for #{name}"
if object
if object.kind_of?(Hash) and object != {}
object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
elsif object.kind_of?(Array) and object != []
object.each_index { |i|
removeBigThings(object[i], "#{name}[#{i}]", event)
}
else
dottedName = name.gsub(/\]\[/, ".").gsub(/\]$/, "").gsub(/^\[/, "")
if dottedName.length > 15 or object.to_s.length > 30
event.remove(name)
end
end
end
end
removeBigThings(event.get("xml"), "[xml]", event)
'
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 16, 2021, 3:25pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/14 "2021-04-16T15:25:49Z")
I do not know what to say. If I add a line
```
puts "Removing #{dottedName}"
```
immediately before the event.remove then I can see it removes everything that meets the condition
```
Called for [xml]
Called for [xml][service]
Called for [xml][service][xmlns]
Removing xml.service.xmlns
Called for [xml][service][content]
Removing xml.service.content
Called for [xml][xmlns:w]
Called for [xml][inputParameter]
Called for [xml][inputParameter][xmlns]
Removing xml.inputParameter.xmlns
```
etc. That leaves
```
"xml" => {
"credential" => {},
"inputParameter" => {
"parameter" => [
[0] {},
[1] {},
[2] {},
[3] {},
[4] {},
[5] {}
]
},
"provider" => {},
"service" => {},
"serviceFunction" => {},
"xmlns:w" => "gk.RiskManagement",
"header" => {}
},
```
---
### Author:  [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)
#### Post date: [April 16, 2021, 4:19pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/15 "2021-04-16T16:19:18Z")
@Badger ,
Really appreciate your quick help every time. it is working as expected.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 14, 2021, 4:19pm UTC](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/16 "2021-05-14T16:19:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.