Hey guys,
I am using http input poller to poll 2 urls, my requirement is to aggregate the JSON replies based on a common field which is "StateFIPS" into a single structure, Here is my data:
Data 1: Give results for all states, value being an array of state data
"State" : [{"StateName": "Maine", "StateFIPS":23, StateCount: 72828} , {"StateName": "New Hampshire", "StateFIPS": 33, StateCount:67733}, {...}]
Data 2: Gives results for all counties, state identified by FIPS code. value being an array of county data
`
"County" : [{"CountyName": "Oxford", "StateFIPS": 23, CountyCount: 328} , {"CountyName": "york", "StateFIPS": 23, CountyCount: 433}, {...}, {"CountyName": "grafton", "StateFIPS": 33, CountyCount: 433}, {...}]
`
Required aggregation: The aggregated result includes info for both state and county. Essentially results equal number of counties included with state info.linked via one nested field i.e. StateFIPS. This data would then be pushed to elasticsearch.
{""StateName": "Maine", "StateFIPS":23, StateCount: 72828, CountyName": "Oxford", CountyCount: 328} , {...}, {...}
I have tried some aggregation examples but can't figure out a way to do this. Any directions/hints/suggestions are appreciable.
My system is Red Hat 4.8.5, logstash version is 6.4.2.