How gsub parse the log line

Elastic Stack Logstash
1

I have a log file in json format having values and corresponding sub-values till 2 levels. All are encapsulated with double quotes. Normally the log is having simple key-value pairs having key and values encapsulated within double quotes but sometimes i get exceptions like given below with sub-values . This is causing parsing issues and i used mutate with gsub like this :

mutate {
gsub => ["json_part", '" """', '" "``' , "json_part", '""" "', '``" "', "json_part", '""', '``']
}

Using above gsub, below lines is not parsed correctly :

"jobName" "do" "command" "env SimuType=rtl VIP_AMBA=0 XPROP=0 TEAL_TARMAC_ENABLE=never xrun -64bit -r DEFAULT_RTL "" """ "ru_utime" "0.860498"

Please note that keys above are jobName, command & ru_utime.

This below log line is parsed correctly with above gsub.
"cpuTime" "14261.006836" "command" """exec"" ""/bin/sh"" ""/home/evinc/F8-80/OPT3/wrapper.sh"" ""1
"" ""0"" ""deckbuild"" ""-noplot"" ""-vwf2"" ""-ascii"" ""-run"" ""/home/evinc/F8-80/OPT3/jobs/3534/3535/3575/3576/3577/deck-ml.in""" "ru_utime" "14261.002999"

keys in above log are cpuTime, command, ru_utime

I want to know how gsub parse the log in case of multiple pattern in the array? Is it from left to right in sqeuence or right to left or if only one matches then gsub stops and applies the replacement string i.e From my given gsub
"json_part", '""', '``' is executed first or "json_part", '" """', '" "``' ?

For me it is not that important to have sub-values. My final requirement is to have maintain key, value and keep/change sub-values encapsulation.

Please suggest.

2

Yes, it is. It will take the first triplet of fieldname / pattern / replacement, make the global substitution, then move on to the next triplet, and so on.

3

Thanks Badger for your answer but i would like to know 2 things :

  1. If i substitute - with ; in first triplet and then substitute ; with - in second triplet. Will i be able to get my original data that exist before gsub?

  2. In my specific case where i have json data encapsulated in double quotes for below log line i.e json_part =
    "jobName" "do" "command" "env SimuType=rtl VIP_AMBA=0 XPROP=0 TEAL_TARMAC_ENABLE=never xrun -64bit -r DEFAULT_RTL "" """ "ru_utime" "0.860498"

if i do gsub
mutate {
gsub => ["json_part", '"" """ "', '" "' ]
}

will the json_part after substitution be able to main it's double quote encapsulation? Will the key and value encapsulation of double quotes be impacted with gsub if i make substitution of double quotes with anything else?

4

Yes.

I did not understand your second question.

5

I apologize for being not clear in my 2nd question.
My json_data with key-value pair is like this in the log file :
"jobName" "do" "command" "env SimuType=rtl VIP_AMBA=0 XPROP=0 TEAL_TARMAC_ENABLE=never xrun -64bit -r DEFAULT_RTL "" """ "ru_utime" "0.860498"

Key Value
"jobName" "do"
"command" "env SimuType=rtl VIP_AMBA=0 XPROP=0 TEAL_TARMAC_ENABLE=never xrun -64bit -r DEFAULT_RTL "" """
"ru_utime" "0.860498"

Now applying following mutate and gsub on above data
mutate {
gsub => ["json_part", '"" """ "', '"' ]
}

The above gsub will search the pattern "" """ " my incoming data and find it between value of "command" key and next key which is "ru_time" here
"jobName" "do" "command" "env SimuType=rtl VIP_AMBA=0 XPROP=0 TEAL_TARMAC_ENABLE=never xrun -64bit -r DEFAULT_RTL **"" """ "**ru_utime" "0.860498"

and it will substitute it with one double quote ". This way either command key-value pair and ru_time key-value pair will loose one encapsulation of double quote and after substitution ru_time will have no double-quote in begin:

Key Value
"jobName" "do"
"command" "env SimuType=rtl VIP_AMBA=0 XPROP=0 TEAL_TARMAC_ENABLE=never xrun -64bit -r DEFAULT_RTL "
**ru_**utime" "0.860498"

I want to understand the behavior of gsub here if gsub will do it like above substitution or it will not be able to find the pattern because it is a json data and should respect the double quote encapsulation for key as well as values.

6

A gsub operation does not know or care whether the string it is operating on is JSON or kv data or anything else.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.