elasticsearch.exceptions.TransportError: TransportError 503: Data too large

Elastic Stack Elasticsearch
1

I'm trying to get the response from ES hitting it from python code but it is showing the below error:

elasticsearch.exceptions.TransportError: TransportError(503, u'search_phase_execution_exception', u'[request] Data too large, data for [<agg [POSCodeModifier]>] would be [623327280/594.4mb], which is larger than the limit of [623326003/594.4mb]')

If i hit the same code from kibana i get the results but using python i'm getting this error. I'm using aggregation in my code. if someone can explain if i need to set some properties or how to optimise it?? Below is the structure for request i'm sending and if i set start and end date greater than 5 days it gives me the error, otherwise i'm getting the results

unmtchd_ESdata= es.search(index='cstore_new',body={'size' : 0, "aggs": {
    "filtered": {
      "filter": {
        "bool": {
          "must_not": [
            {
              "match": {
                "CSPAccountNo": store_id
              }
            }
          ],
          "must": [
            {
              "range": {
                "ReportDate": {
                  "gte": start_dt,
                  "lte": end_dt
                }
              }
            }
          ]
        }
      }
    ,
    "aggs": {
      "POSCode": {
        "terms": {
          "field": "POSCode",
          "size": 10000
        },
        "aggs": {
          "POSCodeModifier": {
            "terms": {
              "field": "POSCodeModifier",
              "size": 10000
            },
            "aggs": {
              "CSP": {
                "terms": {
                  "field": "CSPAccountNo",
                  "size": 10000
                },
                "aggs": {
                  "per_stock": {
                    "date_histogram": {
                      "field": "ReportDate",
                      "interval": "week",
                      "format": "yyyy-MM-dd",
                      "min_doc_count": 0,
                      "extended_bounds": {
                        "min": start_dt,
                        "max": end_dt
                      }
                    },
                    "aggs": {
                      "avg_week_qty_sales": {
                        "sum": {
                          "field": "TotalCount"
                        }
                      }
                    }
                  },
                  "market_week_metrics": {
                    "extended_stats_bucket": {
                      "buckets_path": "per_stock>avg_week_qty_sales"
                    }
                  }
                }
              }
            }

          }
        }
      }
    }
  }
}},request_timeout=1000)
2

Can u show us your python code? Without it optimalisation can be hard :slight_smile:

3

Size:10000 is producing too many data.

4

so which size i should reduce? i declared it at 3 levels in my request and i hope after reducing the size i have all the relevant data??

5

the curl request or kibana hit it self is not working so i don't think you need python code for this

6

You are right. I think you generate request in python.

7

yes using "from elasticsearch import Elasticsearch".

So any suggestions how i can improve on it??

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.