A quick solution would be using gsub (either by mutate filter or ruby filter) on the original message to replace the https: part prior to using KV with something that won't mess with it, like it's encoded value (https%3A) in order to avoid jumping through hoops to deal with it inside the KV filter.
if [clientInfo] =~ /.+/ {
mutate {
gsub => [
# replace in clientInfo the 'https:'
"clientInfo", "https:", "https%3A"
]
My current problem is that I also have Timestamp in the clientInfo.
"clientInfo" => "User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.0 Mobile/14F89 Safari/602.1 :: Refferer(URI):https%3A//wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light :: Client IP:172.58.56.216, 10.10.10.159, 10.108.2.44 ::Timestamp:2017-07-27 13:34:06.329"
To Solve it, I wrote a ruby code, but it doesn't work in the logstash.
(It is running successfully on ruby!)
The clientInfo :
"clientInfo" => "User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.0 Mobile/14F89 Safari/602.1 :: Refferer(URI):https%3A//wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light :: Client IP:172.58.56.216, 10.10.10.159, 10.108.2.44 :: Timestamp:2017-07-27 13:34:06.329"
This is the ruby filter:
ruby {
# Substitute the ":" in the TimeStamp to encoded value.
init => "clientinfostring = event['clientInfo']"
code => " startpoint=@clientinfostring.index("Timestamp")
newstring=@clientinfostring[startpoint+21,8]
updatedstring=newstring.gsub!(":","%3A")
event['clientInfo']=updatedstring"
}
What if you used a Grok filter before the KV filter to pull out the time stamp? Then use include_keys for the KV to only keep what you want from the event, which will not include the "Timestamp" or "34" in the KV filter.
What exactly is the issue with this Ruby code? It doesn't produce the expected results?
Also, why are you defining the clientinfostring on the init block? If I recall correctly, this block is ran only once (on start) so this variable never gets updated with each following event's value. Try moving it inside the actual code block.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.