Overcoming search.max_buckets Limitation in AWS Elasticsearch for Shard-Level Aggregations

Elastic Stack Elasticsearch
1

Objective:

Perform aggregation (terms aggregation) on all documents within each shard.

Actions Taken & Issues:

  • Set size: 100 (assuming size should remain unchanged)
  • Set shard_size to 2147483519
  • Set shard_size to 10000

Result:

The following error occurs:

Trying to create too many buckets. Must be less than or equal to: [65535] but was [65544]. This limit can be set by changing the [search.max_buckets] cluster level setting.

However, the Elasticsearch instance is hosted on AWS (Amazon OpenSearch Service), and modifying search.max_buckets is not possible. https://docs.aws.amazon.com/AmazonS3/latest/userguide/BucketRestrictions.html

Elasticsearch Information:

  • Version: 7.10
  • Shards
JSON[
  {
    "index": "index_name",
    "shard": "0",
    "prirep": "p",
    "state": "STARTED",
    "docs": "50031782",
    "store": "39.3gb",
    ...
  },
  {
    "index": "index_name",
    "shard": "1",
    "prirep": "p",
    "state": "STARTED",
    "docs": "49976007",
    "store": "39.2gb",
    ...
  },
  {
    "index": "index_name",
    "shard": "2",
    "prirep": "p",
    "state": "STARTED",
    "docs": "49976709",
    "store": "39.3gb",
    ...
  },
  {
    "index": "index_name",
    "shard": "3",
    "prirep": "p",
    "state": "STARTED",
    "docs": "49998407",
    "store": "39.3gb",
    ...
  },
  {
    "index": "index_name",
    "shard": "4",
    "prirep": "p",
    "state": "STARTED",
    "docs": "49971995",
    "store": "39.2gb",
    ...
  }
]
2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

This is something you will need to discuss with AWS support as it is a restriction in their service.

4

BTW did you look at Cloud by Elastic, also available if needed from AWS Marketplace, Azure Marketplace and Google Cloud Marketplace?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next :slight_smile: ...