How to map nested json array with http output plugin

Elastic Stack Logstash
1

Hi All

I have incoming events from input as 2 records

Project_Num, Task Name,Task_num,Task Date
PRJ-001,Task Name1,111,1-Jan-2020
PRJ-001,Task Name2,112,2-Jan-2020

How should i map it with my json payload like below in http output
Basically, i need my data in below format to form a payload

There can be a cases where i can have 3 records from same Project
or 4 Records from same project as i need to form a array in json output for payload
So the mapping should be dynamic

Need to group it dynamically to form a one json record

{
    "Project":{
        "Project_Num":"PRJ-001",
		
    "Task":[
        {
            "Task Name":"Task Name1",
            "Task_num":111,
            "Task Date":"1-Jan-2020",
        },
        {
           "Task Name":"Task Name2",
            "Task_num":112,
            "Task Date":"2-Jan-2020",
        }
    ]
}

Thanks

2

Use an aggregate filter.

3

Thanks @Badger for your response, but i am looking some clean solution
I always had issues in aggregate plugin

If you see the case in example 4

When i have data like this

  { "country_name": "France", "town_name": "Paris" }
  { "country_name": "France", "town_name": "Marseille" }
  { "country_name": "USA", "town_name": "New-York" }

this groups perfectly with two records

  { "country_name": "France", "towns": [ {"town_name": "Paris"}, {"town_name": "Marseille"} ] }
  { "country_name": "USA", "towns": [ {"town_name": "New-York"} ] }

But when i have same data jumbled it gives wrong info, thats were aggregate plugin is not reliable

  { "country_name": "France", "town_name": "Paris" }
  { "country_name": "USA", "town_name": "New-York" }
  { "country_name": "France", "town_name": "Marseille" }

If we have data like above, then i get three records.
In my case also the country_name is not necessary will come in sequence always.

Any Idea , how we can map arrays to http output.

4

Mhmh I don't think you can achieve something like this with Logstash. The aggregate filter is what best fits the case but, as you said, if events are not ordered (as often happens) the result is not granted (which is why I rarely use it).

I think your best shot is to organize an entity-centric index using Elasticsearch transformations, but this is not the section where to discuss such stuff.

1 Like
5

So look at example 3, not example 4.

1 Like
6

okay, let me try this . My input data has an end , it is API response in json format

7

great it worked

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.