# How can convert unstructured log string to json format using logstatsh

**URL:** <https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202>\
**Category:** Logstash\
**Created:** [December 20, 2020, 8:29pm UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202 "2020-12-20T20:29:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhi.coerian](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@abhi.coerian](https://discuss.elastic.co/u/abhi.coerian)\
**Post date:** [December 20, 2020, 8:29pm UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/1 "2020-12-20T20:29:18Z")

</div>

INFO handlers.DrivelRequestHandler: 2020-12-14 00:00:15.486 - JOB job\_1603918538928\_4026468 QUEUE queue\_test USER test AUTHORIZED\_SCHEMA message student {\n optional int64 name;\n optional double score;\n required binary roll;\n}

Need to convert into nested json structure :

JOB : job\_1603918538928\_4026468  
user: test  
queue : queue\_test  
table\_name : student  
columns : [  
name,score,roll  
]

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [December 20, 2020, 10:15pm UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/2 "2020-12-20T22:15:01Z")

</div>

you need to use grok filter something like this. here is example for first two

this will give you two field  
job: job\_1603918538928\_4026468  
user: test

grok {  
tag\_on\_failure =\> ["grok\_parse\_failed"]  
match =\> {"message" =\> "%{GREEDYDATA:rm1}- JOB %{WORD:job} %{GREEDYDATA:rm2} USER %{WORD: user}" }  
}  
mutate { remove\_field =\> ["rm1","rm2"] }

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 20, 2020, 10:27pm UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/3 "2020-12-20T22:27:35Z")

</div>

If you do not want to keep a match it is easier not to name it in the first place. Also if you are starting with a field you do not want to keep there is no need to match it:

```
 "- JOB %{WORD:job} %{GREEDYDATA} USER %{WORD: user}"
```

---

<div class="post-metadata">

**Author:** ![abhi.coerian](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@abhi.coerian](https://discuss.elastic.co/u/abhi.coerian)\
**Post date:** [December 21, 2020, 6:18am UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/4 "2020-12-21T06:18:58Z")

</div>

> [@elasticforme](#):
>
> %{GREEDYDATA:rm1}

Thanks !! Its working fine.  
But how I can generate nested json.  
columns : [  
name,score,roll  
]

from {\n optional int64 name;\n optional double score;\n required binary roll;\n}  
NOTE : it could be variable column list

---

<div class="post-metadata">

**Author:** ![abhi.coerian](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@abhi.coerian](https://discuss.elastic.co/u/abhi.coerian)\
**Post date:** [December 23, 2020, 5:29am UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/5 "2020-12-23T05:29:01Z")

</div>

Any suggestion?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [December 23, 2020, 6:45am UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/6 "2020-12-23T06:45:43Z")

</div>

> [@abhi.coerian](#):
>
> {\n optional int64 name;\n optional double score;\n required binary roll;\n}

{\\n %{GREEDYDATA:p1} %{WORD:c1};\\n %{GREEDYDATA:p2} %{WORD:c2};\\n %{GREEDYDATA:p3} %{WORD:c3};\n}

will give you this  
{  
"c3": "roll",  
"p1": "optional int64",  
"p2": "optional double",  
"p3": "required binary",  
"c1": "name",  
"c2": "score"  
}

then you can do mutate to add field called columns  
mutate { add\_field =\> { "columns" =\> "%{c1},%{c2},%{c3}" } }

this will give you columns =\> "name,score,roll" but this is not nested json. it is single field.

if you are asking for columns[0], columns[1] then it is array not nested jason

nested json is I think  
columns { key1: name, key2:score, key3:roll }

then you can refrerence then by columns.key1, columen.key2 etc...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 20, 2021, 6:45am UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202/7 "2021-01-20T06:45:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.