# Terms Aggregation on billions of documents

**URL:** <https://discuss.elastic.co/t/terms-aggregation-on-billions-of-documents/103326>\
**Category:** Elasticsearch\
**Created:** [October 10, 2017, 8:55am UTC](https://discuss.elastic.co/t/terms-aggregation-on-billions-of-documents/103326 "2017-10-10T08:55:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bevans88](https://avatars.discourse-cdn.com/v4/letter/b/a698b9/32.png) [@bevans88](https://discuss.elastic.co/u/bevans88)\
**Post date:** [October 10, 2017, 8:55am UTC](https://discuss.elastic.co/t/terms-aggregation-on-billions-of-documents/103326/1 "2017-10-10T08:55:32Z")

</div>

Hi,

I'm wondering if anyone has any experience with running terms aggregations, with a sub sum aggregation, on a very large index (~100 billion documents). The documents themselves are relatively small (~20 fields). Some of these fields can be high cardinality (~ 40 million).

The requirement here is that we achieve around a 5 second query response. The worse case query for the aggregation is a match\_all.

As far as I'm aware to achieve this scale we'd require a very large cluster, which even then might not achieve the required performance.

Does anyone have any experience/thoughts on this?

Thanks,

Brent

---

<div class="post-metadata">

**Author:** ![bevans88](https://avatars.discourse-cdn.com/v4/letter/b/a698b9/32.png) [@bevans88](https://discuss.elastic.co/u/bevans88)\
**Post date:** [October 11, 2017, 8:02am UTC](https://discuss.elastic.co/t/terms-aggregation-on-billions-of-documents/103326/2 "2017-10-11T08:02:07Z")

</div>

To give some additional information, the mapping for the documents is currently:

```
 {
  "_all" : { "enabled": false },
  "properties" : {
    "field1" : {
      "type" : "text",
      "fielddata": true,
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "eager_global_ordinals": true
        }
      }
    },
    "field2" : {
      "type" : "text",
      "fielddata": true,
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "eager_global_ordinals": true
        }
      }
    },
    "field3" : {
      "type" : "text",
      "fielddata": true,
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "eager_global_ordinals": true
        }
      }
    },
    "field4" : {
      "type" : "text",
      "fielddata": true,
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "eager_global_ordinals": true
        }
      }
    },
    "field5" : {
      "type" : "text",
      "fielddata": true,
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "eager_global_ordinals": true
        }
      }
    },
    "field6" : {
      "type" : "date",
      "format" : "basic_date_time_no_millis"
    },
    "field7" : {
      "type" : "date",
      "format" : "basic_date_time_no_millis"
    },
    "field8" : {
      "type" : "date",
      "format" : "basic_date_time_no_millis"
    },
    "field9" : {
      "type": "text"
    },
    "field10" : {
      "type": "text"
    },
    "field11" : {
      "type" : "text",
      "fielddata": true,
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "eager_global_ordinals": true
        }
      }
    },
    "field12" : {
      "type" : "long"
    }
  }
}

```

And an example aggregation would be:

```
{
  "size" : 0,
  "query" : {
    "match_all" : { }
  },
  "_source" : false,
  "aggregations" : {
    "by_terms" : {
      "terms" : {
        "field" : "field11.keyword",
        "size" : 10000,
        "shard_size" : 1000000,
        "min_doc_count" : 1,
        "shard_min_doc_count" : 0,
        "show_term_doc_count_error" : false,
        "order" : [
          {
            "_count" : "desc"
          },
          {
            "_term" : "asc"
          }
        ]
      },
      "aggregations" : {
        "totalSize" : {
          "sum" : {
            "field" : "size"
          }
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2017, 8:02am UTC](https://discuss.elastic.co/t/terms-aggregation-on-billions-of-documents/103326/3 "2017-11-08T08:02:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.