# How to parse mix json logs

**URL:** <https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594>\
**Category:** Logstash\
**Created:** [February 8, 2019, 9:14am UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594 "2019-02-08T09:14:34Z")\
**Posts on this page:** 9\
**Page:** 2

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 15, 2019, 2:47pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/21 "2019-02-15T14:47:30Z")

</div>

Test the format before parsing...

```
    dissect { mapping => { "message" => "%{ts} %{+ts} | %{restOfLine}" } }
    if [restOfLine] =~ /{.*}/ {
        json { source => "restOfLine" }
    } else if [restOfLine] =~ /\[.*\]/ {
        mutate { gsub => ["restOfLine", "^\[", "", "restOfLine", "\]$", "" ] }
        csv { source => "restOfLine" }
    } else {
        # Handle other format
    }
```

---

<div class="post-metadata">

**Author:** ![amolp](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@amolp](https://discuss.elastic.co/u/amolp)\
**Post date:** [February 16, 2019, 7:57pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/22 "2019-02-16T19:57:14Z")

</div>

Thank you so much for your kind help its working.but i am getting mutiple fields named column in kibana why this is so,could you please explain me.i am adding screen short of it here

 ![col](https://us1.discourse-cdn.com/elastic/original/3X/d/d/dd28617785d9df35de7a13cb7d7a1d3d5b620230.png)

if it is not necessary could we remove this.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 16, 2019, 8:12pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/23 "2019-02-16T20:12:07Z")

</div>

> [@amolp](#):
>
> V4,0833364F,533.330,0,0,533.330,0,0,0,0,-0.849,0,0,-0.849,628.064,0,0,628.064,432.013,  
> 431.847,433.645,430.547,249.423,247.824,251.089,249.355,1.055,0,0,3.164,49.975,38982176.000,0  
> ,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,19/02/03,23:52:12

I assumed you wanted to parse that as a csv. If you do not supply the column names the csv will generate them.

---

<div class="post-metadata">

**Author:** ![amolp](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@amolp](https://discuss.elastic.co/u/amolp)\
**Post date:** [February 16, 2019, 8:20pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/24 "2019-02-16T20:20:21Z")

</div>

ok got it,Thank you Bader.

---

<div class="post-metadata">

**Author:** ![amolp](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@amolp](https://discuss.elastic.co/u/amolp)\
**Post date:** [February 25, 2019, 11:18am UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/25 "2019-02-25T11:18:14Z")

</div>

hi Badger,  
Now I wanted to parse this as csv and want to provide column names for each value which are separated by comma here

V4,0833364F,533.330,0,0,533.330,0,0,0,0,-0.849,0,0,-0.849,628.064,0,0,628.064,432.013,  
431.847,433.645,430.547,249.423,247.824,251.089,249.355,1.055,0,0,3.164,49.975,38982176.000,0  
,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,19/02/03,23:52:1234:

Thank you please help

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 25, 2019, 1:02pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/26 "2019-02-25T13:02:35Z")

</div>

OK, so use the [columns](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-columns) option on the csv filter.

---

<div class="post-metadata">

**Author:** ![amolp](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@amolp](https://discuss.elastic.co/u/amolp)\
**Post date:** [February 25, 2019, 1:45pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/27 "2019-02-25T13:45:15Z")

</div>

Yes I can but the problem is values are not fixed in size they may vary.how do I create cloumn names dynamically as per the number values seperated by comma.

Or do I map only few values with same number of columns like I want to map first two values and last one value which is in date formats,and it should come up with date data type in kibana

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 25, 2019, 1:59pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/28 "2019-02-25T13:59:54Z")

</div>

If you require a different number of columns for different events then you could use a conditional to decide which csv filter to use. Something like

```
if [message] =~ /.*,.*,.*,.*,.*/
    csv {} # events with 5 fields
else if [message] =~ /.*,.*,.*,.*/
    csv {} # events with 4 fields
[...]
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 25, 2019, 2:00pm UTC](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594/29 "2019-03-25T14:00:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/how-to-parse-mix-json-logs/167594.md?page=1)