Logstash json filter not working properly

What does the JSON input look like? And what do you want the result to look like?

"objects in arrays are not well supported" is not an error. It is just a statement of fact.

"requests": [
      {
        "path": null,
        "body": "{\"device\":{\"model\":\"H873\",\"os\":\"Android\",\"devicetype\":1,\"language\":\"en\",\"ua\":\"Mozilla/5.0 (Linux; Android 7.0; LG-H873 Build/NRD90U; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/170.0.0.52.95;]\",\"osv\":\"7.0\",\"ip\":\"99.252.231.212\",\"js\":1,\"make\":\"LG\"},\"ext\":{\"\":\"Y\",\"integration_data\":{\"integration_type\":0}},\"wat\":[\"1\"],\"site\":{\"name\":\"Star\",\"domain\":\"star.com\",\"cat\":[\"IAB12\",\"IAB25-1\"],\"content\":{\"sourcerelationship\":1},\"id\":\"20620\",\"ref\":\"http://jam.com/article/tell-me-pastor/\",\"publisher\":{\"id\":\"12469\"}},\"at\":1,\"imp\":[{\"banner\":{\"btype\":[1,2,4],\"w\":728,\"h\":90,\"id\":\"1\",\"battr\":[8,1,9,10,1018,1034]},\"bidfloorcur\":\"USD\",\"id\":\"1\"}],\"tmax\":85,\"bcat\":[\"IAB8-5\",\"IAB8-18\",\"IAB9-9\",\"IAB102\"],\"id\":\"825353607797991882\",\"badv\":[\"infla.com\"],\"user\":{\"b\":\"821987152133\"}}",

looks like this, I would want each value in some sort of KV format.

What does the configuration look like? Are you using two json filters, like this

filter { json { source => "message" } }
filter { json { source => "[requests][0][body]" } }

i am not, i was trying it like this.
filter {
json {
source => "message"
}
split {
field => "requests"
}
}

How about

    json { source => "message" } 
    split { field => "requests" }
    json { source => "[requests][body]" }

doesnt seem to help

So you successfully parsed one JSON message per request, right? And you dislike the resulting document structure?

Are there any errors in the elasticsearch logs?

right, we have solved this before in the filebeat config, but we are using a script here to send this data. We used json.keys_under_root: true in filebeat to solve this

If you want to move the objects within callouts to the top level you could do it using something similar to this.

ok i will give this a try and let you know. Thanks for the quick responses also!

kv{
source => "message"
value_split => ":"
field_split => ","
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}

This seems to almost have it, is there a way to split on ", rather thaan just the comma. everytime i try logstash breaks.

What is your configuration and in what way does logstash break?

i tried it both ways
kv{
source => "message"
value_split => ":"
field_split_pattern => "\"\,"
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}

kv{

source => "message"
value_split => ":"
field_split => "","
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}

and in what way does logstash break?

the data stops streaming in

for the second option i saw this
The given configuration is invalid. Reason: Expected one of #, {, } at line 10, column 28 (byte 119) after filter {
kv{
source => "message"
value_split_pattern => ""

Use single quotes around this

field_split => '",'

ok that worked , still some tweaks to be done but thank you again for your quick responses! Much appreciated!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.