Fields are merge grok output

Hi himalc :slight_smile:

I think I did not understand your initial expected output then. So, I guess that instead of this:

sessionID result=> 731-ufsN
query_field result=> {"myquery","client","time","total_time"}
query_stats result=>{"select * db_states;","tcp:myhost:12336","10","15"}

You actually expect this output:

sessionID result=> 731-ufsN
query_str => myquery
client => client
execution_time_ms => time
total_time_ms => total_time
query_stats => {"select * db_states;","tcp:myhost:12336","10","15"}
query_field => "myquery" + "client" + "time" + "total_time"

If this is the case, then I guess you need a grok filter AND a mutate filter. The grok would look like this:

%{TIME:timex} %{WORD:Ix} %{NUMBER:nox} (?<code>[^\s]*) %{WORD:stdlog} %{WORD:type} %{NUMBER:numbery} %{NUMBER:noh} %{WORD:dbtype} %{WORD:loguserx} (?<sessionID>[^\s]*) {(?<query_str>[^,]*),(?<client>[^,]*),(?<execution_time_ms>[^,]*),(?<total_time_ms>[^}]*)}{(?<query_stats>[^}]*)}$

Then, you would need to "construct" the query_stats field, and this is where you use the "mutate" filter. I can't test it right now, but this post could help you Adding a field from existing ones.

The formula I'm using in the grok is quite easy and is always the same. Basically I use some literals to "pinpoint" somethings (look at the commas for instance, those are just literal matches).
Then I use this extended group (?subexp) over and over again :slight_smile: You can see more info about that here: https://github.com/kkos/oniguruma/blob/master/doc/RE

Then is only a regular expression where I just match "any character until (^) the character which in most cases is a comma. Then I just use a quantifier () so I match "all the characters" until the negation. For this you'll see "(?<field_name>[^,]).

Hope this helped you.