Value split is ":"

Hi,

I have kv text with the value split ":". In the text I also have some ":".

For example : (after the https.)
Refferer(URI):https://wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light

My filter looks like that:

        if [clientInfo] =~ /.+/ {
            kv { 
                 value_split => ":"
                 field_split => "::"
                 source => "clientInfo"
            }
        }

The output result is :

   ` " R\nefferer(URI)" => "https",`

How can I handle it?

Regards,
Sharon.

A quick solution would be using gsub (either by mutate filter or ruby filter) on the original message to replace the https: part prior to using KV with something that won't mess with it, like it's encoded value (https%3A) in order to avoid jumping through hoops to deal with it inside the KV filter.

great.

I used mutate and it worked.

       if [clientInfo] =~ /.+/ {
           mutate {
                   gsub => [
                         # replace in clientInfo the 'https:'
                         "clientInfo", "https:", "https%3A"
                   ]

My current problem is that I also have Timestamp in the clientInfo.

  "clientInfo" => "User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.0 Mobile/14F89 Safari/602.1 :: Refferer(URI):https%3A//wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light :: Client IP:172.58.56.216, 10.10.10.159, 10.108.2.44 ::Timestamp:2017-07-27 13:34:06.329"

the kvstring command break it wrongly too:

"Timestamp" => "2017-07-27 13",

"34" => "06.329",

Thanks
Sharon.

To Solve it, I wrote a ruby code, but it doesn't work in the logstash.
(It is running successfully on ruby!)

The clientInfo :

 "clientInfo" => "User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.0 Mobile/14F89 Safari/602.1 :: Refferer(URI):https%3A//wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light :: Client IP:172.58.56.216, 10.10.10.159, 10.108.2.44 :: Timestamp:2017-07-27 13:34:06.329"

This is the ruby filter:

           ruby {
                  # Substitute the ":" in the TimeStamp to encoded value.
                  init => "clientinfostring = event['clientInfo']"
                  code => " startpoint=@clientinfostring.index("Timestamp")
                            newstring=@clientinfostring[startpoint+21,8]
                            updatedstring=newstring.gsub!(":","%3A")
                            event['clientInfo']=updatedstring"
           }

Any idea?

Thanks
Sharon.

What if you used a Grok filter before the KV filter to pull out the time stamp? Then use include_keys for the KV to only keep what you want from the event, which will not include the "Timestamp" or "34" in the KV filter.

Timestamp:%{TIMESTAMP_ISO8601}

https://grokdebug.herokuapp.com/

What exactly is the issue with this Ruby code? It doesn't produce the expected results?

Also, why are you defining the clientinfostring on the init block? If I recall correctly, this block is ran only once (on start) so this variable never gets updated with each following event's value. Try moving it inside the actual code block.

It actually doesn't produce nothing from the Logstash.
I will try to move it into the code block.

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.