How can convert unstructured log string to json format using logstatsh

[abhi.coerian (Abhinav)]

INFO handlers.DrivelRequestHandler: 2020-12-14 00:00:15.486 - JOB job_1603918538928_4026468 QUEUE queue_test USER test AUTHORIZED_SCHEMA message student {\n optional int64 name;\n optional double score;\n required binary roll;\n}

Need to convert into nested json structure :

JOB : job_1603918538928_4026468
user: test
queue : queue_test
table_name : student
columns : [
name,score,roll
]

[elasticforme (Sachin)]

you need to use grok filter something like this. here is example for first two

this will give you two field
job: job_1603918538928_4026468
user: test

grok {
tag_on_failure => ["grok_parse_failed"]
match => {"message" => "%{GREEDYDATA:rm1}- JOB %{WORD:job} %{GREEDYDATA:rm2} USER %{WORD: user}" }
}
mutate { remove_field => [ "rm1","rm2"] }

[Badger]

If you do not want to keep a match it is easier not to name it in the first place. Also if you are starting with a field you do not want to keep there is no need to match it:

 "- JOB %{WORD:job} %{GREEDYDATA} USER %{WORD: user}"

[abhi.coerian (Abhinav)]

@elasticforme

%{GREEDYDATA:rm1}

Thanks !! Its working fine.
But how I can generate nested json.
columns : [
name,score,roll
]

from {\n optional int64 name;\n optional double score;\n required binary roll;\n}
NOTE : it could be variable column list

[abhi.coerian (Abhinav)]

Any suggestion?

[elasticforme (Sachin)]

@abhi.coerian

{\n optional int64 name;\n optional double score;\n required binary roll;\n}

{\\n %{GREEDYDATA:p1} %{WORD:c1};\\n %{GREEDYDATA:p2} %{WORD:c2};\\n %{GREEDYDATA:p3} %{WORD:c3};\n}

will give you this
{
"c3": "roll",
"p1": "optional int64",
"p2": "optional double",
"p3": "required binary",
"c1": "name",
"c2": "score"
}

then you can do mutate to add field called columns
mutate { add_field => { "columns" => "%{c1},%{c2},%{c3}" } }

this will give you columns => "name,score,roll" but this is not nested json. it is single field.

if you are asking for columns[0], columns[1] then it is array not nested jason

nested json is I think
columns { key1: name, key2:score, key3:roll }

then you can refrerence then by columns.key1, columen.key2 etc...