# Logstash json filter not working properly

**URL:** <https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227>\
**Category:** Logstash\
**Created:** [August 13, 2018, 7:09pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227 "2018-08-13T19:09:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 13, 2018, 7:09pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/1 "2018-08-13T19:09:53Z")

</div>

I am using all the Latest version of ELK , we are trying to parse json using the json filter. some of the data gets filtered properly but other fields have an error saying "objects in arrays are not well supported" Which then the json data is NOT broken up in KV format. Any idea what can be causing this or how to fix it?

Thanks ,  
Kenneth

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2018, 7:38pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/2 "2018-08-13T19:38:04Z")

</div>

What does the JSON input look like? And what do you want the result to look like?

"objects in arrays are not well supported" is not an error. It is just a statement of fact.

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 13, 2018, 7:46pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/3 "2018-08-13T19:46:13Z")

</div>

```
"requests": [
      {
        "path": null,
        "body": "{\"device\":{\"model\":\"H873\",\"os\":\"Android\",\"devicetype\":1,\"language\":\"en\",\"ua\":\"Mozilla/5.0 (Linux; Android 7.0; LG-H873 Build/NRD90U; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/170.0.0.52.95;]\",\"osv\":\"7.0\",\"ip\":\"99.252.231.212\",\"js\":1,\"make\":\"LG\"},\"ext\":{\"\":\"Y\",\"integration_data\":{\"integration_type\":0}},\"wat\":[\"1\"],\"site\":{\"name\":\"Star\",\"domain\":\"star.com\",\"cat\":[\"IAB12\",\"IAB25-1\"],\"content\":{\"sourcerelationship\":1},\"id\":\"20620\",\"ref\":\"http://jam.com/article/tell-me-pastor/\",\"publisher\":{\"id\":\"12469\"}},\"at\":1,\"imp\":[{\"banner\":{\"btype\":[1,2,4],\"w\":728,\"h\":90,\"id\":\"1\",\"battr\":[8,1,9,10,1018,1034]},\"bidfloorcur\":\"USD\",\"id\":\"1\"}],\"tmax\":85,\"bcat\":[\"IAB8-5\",\"IAB8-18\",\"IAB9-9\",\"IAB102\"],\"id\":\"825353607797991882\",\"badv\":[\"infla.com\"],\"user\":{\"b\":\"821987152133\"}}",

```

looks like this, I would want each value in some sort of KV format.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2018, 8:11pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/4 "2018-08-13T20:11:27Z")

</div>

What does the configuration look like? Are you using two json filters, like this

```
filter { json { source => "message" } }
filter { json { source => "[requests][0][body]" } }
```

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 13, 2018, 8:14pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/5 "2018-08-13T20:14:05Z")

</div>

i am not, i was trying it like this.  
filter {  
json {  
source =\> "message"  
}  
split {  
field =\> "requests"  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2018, 8:29pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/6 "2018-08-13T20:29:34Z")

</div>

How about

```
    json { source => "message" } 
    split { field => "requests" }
    json { source => "[requests][body]" }
```

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 13, 2018, 8:44pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/7 "2018-08-13T20:44:46Z")

</div>

doesnt seem to help

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 13, 2018, 8:56pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/9 "2018-08-13T20:56:10Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac0d4c7e392105315be5cced6c61d9cfaad3d749.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2018, 9:16pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/10 "2018-08-13T21:16:24Z")

</div>

So you successfully parsed one JSON message per request, right? And you dislike the resulting document structure?

Are there any errors in the elasticsearch logs?

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 13, 2018, 9:25pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/11 "2018-08-13T21:25:23Z")

</div>

right, we have solved this before in the filebeat config, but we are using a script here to send this data. We used json.keys\_under\_root: true in filebeat to solve this

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2018, 10:10pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/12 "2018-08-13T22:10:32Z")

</div>

If you want to move the objects within callouts to the top level you could do it using something similar to [this](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/2?u=badger).

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 14, 2018, 1:38pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/13 "2018-08-14T13:38:11Z")

</div>

ok i will give this a try and let you know. Thanks for the quick responses also!

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 15, 2018, 5:01pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/14 "2018-08-15T17:01:43Z")

</div>

kv{  
source =\> "message"  
value\_split =\> ":"  
field\_split =\> ","  
include\_brackets =\> false  
recursive =\> "true"  
remove\_char\_value =\> "{}[]"  
remove\_char\_key =\> "{}[]"  
}  
}

This seems to almost have it, is there a way to split on ", rather thaan just the comma. everytime i try logstash breaks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 15, 2018, 5:06pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/15 "2018-08-15T17:06:00Z")

</div>

> [@kmroz](#):
>
> everytime i try logstash breaks

What is your configuration and in what way does logstash break?

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 15, 2018, 5:11pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/16 "2018-08-15T17:11:41Z")

</div>

i tried it both ways  
kv{  
source =\> "message"  
value\_split =\> ":"  
field\_split\_pattern =\> "\"\,"  
include\_brackets =\> false  
recursive =\> "true"  
remove\_char\_value =\> "{}[]"  
remove\_char\_key =\> "{}[]"  
}  
}

```
kv{

```

source =\> "message"  
value\_split =\> ":"  
field\_split =\> "","  
include\_brackets =\> false  
recursive =\> "true"  
remove\_char\_value =\> "{}[]"  
remove\_char\_key =\> "{}[]"  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 15, 2018, 5:13pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/17 "2018-08-15T17:13:05Z")

</div>

and in what way does logstash break?

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 15, 2018, 5:14pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/18 "2018-08-15T17:14:26Z")

</div>

the data stops streaming in

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 15, 2018, 5:15pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/19 "2018-08-15T17:15:16Z")

</div>

for the second option i saw this  
The given configuration is invalid. Reason: Expected one of #, {, } at line 10, column 28 (byte 119) after filter {  
kv{  
source =\> "message"  
value\_split\_pattern =\> ""

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 15, 2018, 5:20pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/20 "2018-08-15T17:20:04Z")

</div>

> [@kmroz](#):
>
> field\_split =\> "","

Use single quotes around this

```
field_split => '",'

```

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [August 15, 2018, 5:24pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227/21 "2018-08-15T17:24:49Z")

</div>

ok that worked , still some tweaks to be done but thank you again for your quick responses! Much appreciated!

[Next page](https://discuss.elastic.co/t/logstash-json-filter-not-working-properly/144227.md?page=2)