How can convert unstructured log string to json format using logstatsh

INFO handlers.DrivelRequestHandler: 2020-12-14 00:00:15.486 - JOB job_1603918538928_4026468 QUEUE queue_test USER test AUTHORIZED_SCHEMA message student {\n optional int64 name;\n optional double score;\n required binary roll;\n}

Need to convert into nested json structure :

JOB : job_1603918538928_4026468
user: test
queue : queue_test
table_name : student
columns : [

you need to use grok filter something like this. here is example for first two

this will give you two field
job: job_1603918538928_4026468
user: test

grok {
tag_on_failure => ["grok_parse_failed"]
match => {"message" => "%{GREEDYDATA:rm1}- JOB %{WORD:job} %{GREEDYDATA:rm2} USER %{WORD: user}" }
mutate { remove_field => [ "rm1","rm2"] }

If you do not want to keep a match it is easier not to name it in the first place. Also if you are starting with a field you do not want to keep there is no need to match it:

 "- JOB %{WORD:job} %{GREEDYDATA} USER %{WORD: user}"

Thanks !! Its working fine.
But how I can generate nested json.
columns : [

from {\n optional int64 name;\n optional double score;\n required binary roll;\n}
NOTE : it could be variable column list

Any suggestion?

{\\n %{GREEDYDATA:p1} %{WORD:c1};\\n %{GREEDYDATA:p2} %{WORD:c2};\\n %{GREEDYDATA:p3} %{WORD:c3};\n}

will give you this
"c3": "roll",
"p1": "optional int64",
"p2": "optional double",
"p3": "required binary",
"c1": "name",
"c2": "score"

then you can do mutate to add field called columns
mutate { add_field => { "columns" => "%{c1},%{c2},%{c3}" } }

this will give you columns => "name,score,roll" but this is not nested json. it is single field.

if you are asking for columns[0], columns[1] then it is array not nested jason

nested json is I think
columns { key1: name, key2:score, key3:roll }

then you can refrerence then by columns.key1, columen.key2 etc...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.