What does the JSON input look like? And what do you want the result to look like?
"objects in arrays are not well supported" is not an error. It is just a statement of fact.
What does the JSON input look like? And what do you want the result to look like?
"objects in arrays are not well supported" is not an error. It is just a statement of fact.
"requests": [
{
"path": null,
"body": "{\"device\":{\"model\":\"H873\",\"os\":\"Android\",\"devicetype\":1,\"language\":\"en\",\"ua\":\"Mozilla/5.0 (Linux; Android 7.0; LG-H873 Build/NRD90U; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/170.0.0.52.95;]\",\"osv\":\"7.0\",\"ip\":\"99.252.231.212\",\"js\":1,\"make\":\"LG\"},\"ext\":{\"\":\"Y\",\"integration_data\":{\"integration_type\":0}},\"wat\":[\"1\"],\"site\":{\"name\":\"Star\",\"domain\":\"star.com\",\"cat\":[\"IAB12\",\"IAB25-1\"],\"content\":{\"sourcerelationship\":1},\"id\":\"20620\",\"ref\":\"http://jam.com/article/tell-me-pastor/\",\"publisher\":{\"id\":\"12469\"}},\"at\":1,\"imp\":[{\"banner\":{\"btype\":[1,2,4],\"w\":728,\"h\":90,\"id\":\"1\",\"battr\":[8,1,9,10,1018,1034]},\"bidfloorcur\":\"USD\",\"id\":\"1\"}],\"tmax\":85,\"bcat\":[\"IAB8-5\",\"IAB8-18\",\"IAB9-9\",\"IAB102\"],\"id\":\"825353607797991882\",\"badv\":[\"infla.com\"],\"user\":{\"b\":\"821987152133\"}}",
looks like this, I would want each value in some sort of KV format.
What does the configuration look like? Are you using two json filters, like this
filter { json { source => "message" } }
filter { json { source => "[requests][0][body]" } }
i am not, i was trying it like this.
filter {
json {
source => "message"
}
split {
field => "requests"
}
}
How about
json { source => "message" }
split { field => "requests" }
json { source => "[requests][body]" }
doesnt seem to help
So you successfully parsed one JSON message per request, right? And you dislike the resulting document structure?
Are there any errors in the elasticsearch logs?
right, we have solved this before in the filebeat config, but we are using a script here to send this data. We used json.keys_under_root: true in filebeat to solve this
If you want to move the objects within callouts to the top level you could do it using something similar to this.
ok i will give this a try and let you know. Thanks for the quick responses also!
kv{
source => "message"
value_split => ":"
field_split => ","
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}
This seems to almost have it, is there a way to split on ", rather thaan just the comma. everytime i try logstash breaks.
What is your configuration and in what way does logstash break?
i tried it both ways
kv{
source => "message"
value_split => ":"
field_split_pattern => "\"\,"
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}
kv{
source => "message"
value_split => ":"
field_split => "","
include_brackets => false
recursive => "true"
remove_char_value => "{}[]"
remove_char_key => "{}[]"
}
}
and in what way does logstash break?
the data stops streaming in
for the second option i saw this
The given configuration is invalid. Reason: Expected one of #, {, } at line 10, column 28 (byte 119) after filter {
kv{
source => "message"
value_split_pattern => ""
Use single quotes around this
field_split => '",'
ok that worked , still some tweaks to be done but thank you again for your quick responses! Much appreciated!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.