Value split is ":"


(Sharon Sasporta) #1

Hi,

I have kv text with the value split ":". In the text I also have some ":".

For example : (after the https.)
Refferer(URI):https://wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light

My filter looks like that:

        if [clientInfo] =~ /.+/ {
            kv { 
                 value_split => ":"
                 field_split => "::"
                 source => "clientInfo"
            }
        }

The output result is :

   ` " R\nefferer(URI)" => "https",`

How can I handle it?

Regards,
Sharon.


(Paris Mermigkas) #2

A quick solution would be using gsub (either by mutate filter or ruby filter) on the original message to replace the https: part prior to using KV with something that won't mess with it, like it's encoded value (https%3A) in order to avoid jumping through hoops to deal with it inside the KV filter.


(Sharon Sasporta) #3

great.

I used mutate and it worked.

       if [clientInfo] =~ /.+/ {
           mutate {
                   gsub => [
                         # replace in clientInfo the 'https:'
                         "clientInfo", "https:", "https%3A"
                   ]

My current problem is that I also have Timestamp in the clientInfo.

  "clientInfo" => "User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.0 Mobile/14F89 Safari/602.1 :: Refferer(URI):https%3A//wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light :: Client IP:172.58.56.216, 10.10.10.159, 10.108.2.44 ::Timestamp:2017-07-27 13:34:06.329"

the kvstring command break it wrongly too:

"Timestamp" => "2017-07-27 13",

"34" => "06.329",

Thanks
Sharon.


(Sharon Sasporta) #4

To Solve it, I wrote a ruby code, but it doesn't work in the logstash.
(It is running successfully on ruby!)

The clientInfo :

 "clientInfo" => "User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.0 Mobile/14F89 Safari/602.1 :: Refferer(URI):https%3A//wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui_style=light :: Client IP:172.58.56.216, 10.10.10.159, 10.108.2.44 :: Timestamp:2017-07-27 13:34:06.329"

This is the ruby filter:

           ruby {
                  # Substitute the ":" in the TimeStamp to encoded value.
                  init => "clientinfostring = event['clientInfo']"
                  code => " startpoint=@clientinfostring.index("Timestamp")
                            newstring=@clientinfostring[startpoint+21,8]
                            updatedstring=newstring.gsub!(":","%3A")
                            event['clientInfo']=updatedstring"
           }

Any idea?

Thanks
Sharon.


#5

What if you used a Grok filter before the KV filter to pull out the time stamp? Then use include_keys for the KV to only keep what you want from the event, which will not include the "Timestamp" or "34" in the KV filter.

Timestamp:%{TIMESTAMP_ISO8601}

https://grokdebug.herokuapp.com/


(Paris Mermigkas) #6

What exactly is the issue with this Ruby code? It doesn't produce the expected results?

Also, why are you defining the clientinfostring on the init block? If I recall correctly, this block is ran only once (on start) so this variable never gets updated with each following event's value. Try moving it inside the actual code block.


(Sharon Sasporta) #7

It actually doesn't produce nothing from the Logstash.
I will try to move it into the code block.

Thanks


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.