Problem when splitting a nested JSON array

Elastic Stack Elasticsearch
1

Hi,

I have seen multiple thread talking about this problem but with some differences. I have the following log schema:

"messageId": 123,
"timestamp": "2020-09-14T03:35:37",
"extra": {
    "name": "typefood",
    "food_changes": [ {
         "property": "coca",
         "data_type": "experience"},
          {"property": "tea",
          "data_type": "amazing"
          }
    ]

I would like to get what you can see on "food_changes" splitted into two different logs like this outcome:

    "message.id": 123,
    "timestamp": "2020-09-14T03:35:37",
    "extra.name": "typefood"
    "extra.food_changes.property": "coca"
    "extra.food_changes.data_type": "experience"

    "message.id": 123,
    "timestamp": "2020-09-14T03:35:37",
    "extra.name": "typefood"
    "extra.food_changes.property": "tea"
    "extra.food_changes.data_type": "amazing"

What I am trying is:
filter {
json {
source => "message"
}
split {
field => "[extra][food_changes]"
}
mutate {
add_field =>
"[message][id]" => "%{[messageid]}

}

But it only shows in Kibana the fields for the first log...

"messageid": 123,
"timestamp": "2020-09-14T03:35:37",
"extra.name": "typefood"
"extra.food_changes.property": "coca"
"extra.food_changes.data_type": "experience"

not for the second log...

"messageid": 123,
"timestamp": "2020-09-14T03:35:37",
"extra.name": "typefood"
"extra.food_changes.property": "tea"
"extra.food_changes.data_type": "amazing"

Can someone please give me some insight?

2

What is your full logstash pipeline? Are you setting document_id to something or are you letting elasticsearch set the document id?

3

Hi Leandro,

@leandrojmp . Yes, I am letting elasticsearch to set the document id.

with "full logstash pipeline", do you mean the logstash.yml or where my config to ingest the logs I pasted before is?

4

Your full pipeline configuration, you shared only part of it, you need to share the complete pipeline with the inputs, filters and outputs.

5

Hi @leandrojmp,

Thank you very much for your willingness to help. I made some changes compared to what I saw here and it worked. The problem of my real log schema is that it has few nested JSON arrays at different levels that's why it was a bit more complex than what you can see here:

Thank you Leandro again.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.