Python Elasticsearch aggregated query

[Sederfo (Vlad)]

I am struggling to get an aggregated response based on a query... for example, in Kibana visualization, after creating a table with the aggregations and query I need, I can view the request by going inside the visualization and Inspect -> View:Requests -> Request. This gives me the entire request, structured like this:

{
 "aggs":{...},
  "size": 0,
  "fields":[...],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool":{...}
  }
}

How can i obtain the same dataset that is displayed in Kibana Visualization with the API?
I tried translating the API Request fields to the es.search() function like this:

result = es.search(
index=index,
fields=[ ...],
script_fields={},
stored_fields=[
    "*"
],
runtime_mappings={},
_source={
    "excludes": []
},
query={...},
aggs={...},
size = 0,
scroll = '5m'
)

But the data is not aggregated correctly, like in the Kibana Visualization.
Another strange behaviour is that if i completely remove the aggs = {...} from es.search(), it gives me the same dataset.

[majagrubic (Maja)]

Could you paste the full aggs part of the request from Kibana visualization, as well as the one you are trying to make, and the ES response, please?

[Sederfo (Vlad)]

"aggs": {
    "0": {
      "terms": {
        "field": "source.ip",
        "order": {
          "_count": "desc"
        },
        "size": 200
      },
      "aggs": {
        "1": {
          "terms": {
            "field": "destination.ip",
            "order": {
              "_count": "desc"
            },
            "size": 200
          },
          "aggs": {
            "2": {
              "terms": {
                "field": "device.hostname",
                "order": {
                  "_count": "desc"
                },
                "size": 3
              },
              "aggs": {
                "4": {
                  "sum": {
                    "field": "source.packets"
                  }
                },
                "5": {
                  "sum": {
                    "field": "destination.packets"
                  }
                },
                "6": {
                  "sum": {
                    "field": "source.bytes"
                  }
                },
                "7": {
                  "sum": {
                    "field": "destination.bytes"
                  }
                }
              }
            }
          }
        }
      }
    }
  }

I passed this dictionary type object to the 'aggs' (also tried 'aggregations') parameter of the es.search() function.

I cannot post the whole ES response since it is connection events from our firewalls
The ES Reponse looks like this:

"total": {
        "value": 74,
        "relation": "eq"
    },
    "max_score": 0.0,
    "hits":[...]

And the documents are in the hits list, without any kind of aggregation applied.
This aggregation is supposed to group source and destination ip's and sum their packets and bytes.

[Sederfo (Vlad)]

This is the format of the data displayed in Kibana after applying the aggregations and this is the result I need (either json or csv or whatever as long as I manage to get this)
Top values of source.ip

Top values of destination.ip

Top values of device.hostname

Count of records

Sum of source.packets

Sum of destination.packets

Sum of source.bytes

Sum of destination.bytes

[majagrubic (Maja)]

The request seems correct. The parameter for the request is indeed called aggs, and the ES will return aggregations under aggregations section of the response. Please note that this will be separate from the hits section. So you should have something like this in your ES response:

"aggregations" : {
    "0" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ ]
    },
    "1" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "...",
          "doc_count" : 2885
        },
        {
          "key" : "...",
          "doc_count" : 2825
        },
        ....
      ]
    }
  }

[Sederfo (Vlad)]

Thank you so so much! I completely missed that!