Logstash json filter not working properly

Elastic Stack Logstash
1

I am using all the Latest version of ELK , we are trying to parse json using the json filter. some of the data gets filtered properly but other fields have an error saying "objects in arrays are not well supported" Which then the json data is NOT broken up in KV format. Any idea what can be causing this or how to fix it?

Thanks ,
Kenneth

2

What does the JSON input look like? And what do you want the result to look like?

"objects in arrays are not well supported" is not an error. It is just a statement of fact.

3 
"requests": [
      {
        "path": null,
        "body": "{\"device\":{\"model\":\"H873\",\"os\":\"Android\",\"devicetype\":1,\"language\":\"en\",\"ua\":\"Mozilla/5.0 (Linux; Android 7.0; LG-H873 Build/NRD90U; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/170.0.0.52.95;]\",\"osv\":\"7.0\",\"ip\":\"99.252.231.212\",\"js\":1,\"make\":\"LG\"},\"ext\":{\"\":\"Y\",\"integration_data\":{\"integration_type\":0}},\"wat\":[\"1\"],\"site\":{\"name\":\"Star\",\"domain\":\"star.com\",\"cat\":[\"IAB12\",\"IAB25-1\"],\"content\":{\"sourcerelationship\":1},\"id\":\"20620\",\"ref\":\"http://jam.com/article/tell-me-pastor/\",\"publisher\":{\"id\":\"12469\"}},\"at\":1,\"imp\":[{\"banner\":{\"btype\":[1,2,4],\"w\":728,\"h\":90,\"id\":\"1\",\"battr\":[8,1,9,10,1018,1034]},\"bidfloorcur\":\"USD\",\"id\":\"1\"}],\"tmax\":85,\"bcat\":[\"IAB8-5\",\"IAB8-18\",\"IAB9-9\",\"IAB102\"],\"id\":\"825353607797991882\",\"badv\":[\"infla.com\"],\"user\":{\"b\":\"821987152133\"}}",

looks like this, I would want each value in some sort of KV format.

4

What does the configuration look like? Are you using two json filters, like this

filter { json { source => "message" } }
filter { json { source => "[requests][0][body]" } }
5

i am not, i was trying it like this.
filter {
json {
source => "message"
}
split {
field => "requests"
}
}

6

How about

    json { source => "message" } 
    split { field => "requests" }
    json { source => "[requests][body]" }
7

doesnt seem to help

9

image
image.png1852×250 18.5 KB

10

So you successfully parsed one JSON message per request, right? And you dislike the resulting document structure?

Are there any errors in the elasticsearch logs?

11

right, we have solved this before in the filebeat config, but we are using a script here to send this data. We used json.keys_under_root: true in filebeat to solve this

12

If you want to move the objects within callouts to the top level you could do it using something similar to this.

13

ok i will give this a try and let you know. Thanks for the quick responses also!

14

kv{
source => "message"
value_split => ":"
field_split => ","
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}

This seems to almost have it, is there a way to split on ", rather thaan just the comma. everytime i try logstash breaks.

15

What is your configuration and in what way does logstash break?

16

i tried it both ways
kv{
source => "message"
value_split => ":"
field_split_pattern => "\"\,"
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}

kv{

source => "message"
value_split => ":"
field_split => "","
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}

17

and in what way does logstash break?

18

the data stops streaming in

19

for the second option i saw this
The given configuration is invalid. Reason: Expected one of #, {, } at line 10, column 28 (byte 119) after filter {
kv{
source => "message"
value_split_pattern => ""

20

Use single quotes around this

field_split => '",'
21

ok that worked , still some tweaks to be done but thank you again for your quick responses! Much appreciated!