I have an index which contains millions of documents that I want to aggregate.
This is the query I am using to obtain the aggregated results.
aggs = {
"0": {
"terms": {
"field": "source.ip",
"order": {
"_count": "desc"
},
"include": {
"partition": i,
"num_partitions": nr_partitions
},
"size": 1000
},
"aggs": {
"1": {
"terms": {
"field": "destination.ip",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"2": {
"terms": {
"field": "destination.port",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"3": {
"terms": {
"field": "cisco.ftd.security.protocol",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"4": {
"terms": {
"field": "cisco.ftd.security.ingress_interface",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"5": {
"terms": {
"field": "cisco.ftd.security.egress_interface",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"6": {
"terms": {
"field": "cisco.ftd.security.ingress_zone",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"7": {
"terms": {
"field": "cisco.ftd.security.egress_zone",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"8": {
"terms": {
"field": "cisco.ftd.security.access_control_rule_name",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"9": {
"terms": {
"field": "url.original",
"missing": "N/A",
"order": {
"_count": "desc"
},
"size": 1000
},
"aggs": {
"11": {
"sum": {
"field": "source.packets"
}
},
"12": {
"sum": {
"field": "destination.packets"
}
},
"13": {
"sum": {
"field": "source.bytes"
}
},
"14": {
"sum": {
"field": "destination.bytes"
}
},
"15": {
"avg": {
"field": "source.packets"
}
},
"16": {
"avg": {
"field": "destination.packets"
}
},
"17": {
"avg": {
"field": "source.bytes"
}
},
"18": {
"avg": {
"field": "destination.bytes"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
If there are too many documents in the time range that I am querying for, the API throws search_phase_execution_exception with a 503 status code.
How can I obtain the aggregated results for the millions of documents? I tried increasing the number of partitions, but it still gives the same exception.