Number of Shards for an Index

Elastic Stack Elasticsearch
1

Hi,
We are trying to come up with optimum number of shards for the Index. Based on https://www.elastic.co/guide/en/elasticsearch/guide/current/capacity-planning.html, we have done load testing and found that 750K documents per shard will be ideal. However, when we calculated the potential number of documents that could be indexed, we arrive at ~350 shards per index. Based on other internet searches, this seems to be high number.

There are few pages suggested that ideal shard count would be 1 shard /node. We would have ~300M documents in the index with each document of <1KB size. Could anyone please help with determining optimum number of shards/index?

Thanks in advance.

2

Can you explain how you did your testing and arrived at the 750K number?

3

Thanks for responding. My colleague had loaded the data in single node and single shard environment as suggested in the https://www.elastic.co/guide/en/elasticsearch/guide/current/capacity-planning.html , loaded real documents and ran the aggregation queries that we would run to see the response time. Based on the response time, we believe that 750K docs for a shard could be ideal number.

please suggest if there are alternate ways.

4

Any more info you can share? What sort of agg, what sort of docs? How are you defining the SLA?

5

HI Mark,
Please find the details below.

Below is the mapping.

PUT /weekly_performance
{
  "settings": {
    "number_of_shards": 36,
    "number_of_replicas": 1,
    "index.routing.allocation.total_shards_per_node": 24
  },
  "mappings": {
    "weekly_performance": {
      "_all": {
        "enabled": false
      },
      "properties": {
        "category1_id": {
          "type": "integer"
        },
        "category2_id": {
          "type": "integer"
        },
        "category3_id": {
          "type": "integer"
        },
        "category2_name": {
          "type": "text",
          "index": "not_analyzed"
        },
        "category4_id": {
          "type": "text",
          "index": "not_analyzed"
        },
        "category5_id": {
          "type": "text",
          "index": "not_analyzed"
        },
        "category6_id": {
          "type": "integer"
        },
        "category6_alt_id": {
          "type": "text",
          "index": "not_analyzed"
        },
        "location_id": {
          "type": "text",
          "index": "not_analyzed"
        },
        "location_role": {
          "type": "text",
          "index": "not_analyzed"
        },
        "location_channel": {
          "type": "text",
          "index": "not_analyzed"
        },
        "actual_location_id": {
          "type": "text",
          "index": "not_analyzed"
        },
        "actual_location_role": {
          "type": "text",
          "index": "not_analyzed"
        },
        "actual_location_channel": {
          "type": "text",
          "index": "not_analyzed"
        },
        "performance_code": {
          "type": "integer"
        },
        "week_end_date": {
          "type": "date",
          "index": "not_analyzed"
        },
        "week_id": {
          "type": "text",
          "index": "not_analyzed"
        },
        "performance_unit_1": {
          "type": "double"
        },
        "performance_a_amount_1": {
          "type": "double"
        },
        "performance_b_amount_1": {
          "type": "double"
        },
        "performance_unit_2": {
          "type": "double"
        },
        "performance_a_amount_2": {
          "type": "double"
        },
        "performance_b_amount_2": {
          "type": "double"
        },
        "performance_unit_3": {
          "type": "double"
        },
        "performance_a_amount_3": {
          "type": "double"
        },
        "performance_b_amount_3": {
          "type": "double"
        },
        "performance_unit_4": {
          "type": "double"
        },
        "performance_a_amount_4": {
          "type": "double"
        },
        "performance_b_amount_4": {
          "type": "double"
        },
        "performance_unit_5": {
          "type": "double"
        },
        "performance_a_amount_5": {
          "type": "double"
        },
        "performance_b_amount_5": {
          "type": "double"
        },
        "performance_unit_6": {
          "type": "double"
        },
        "performance_a_amount_6": {
          "type": "double"
        },
        "performance_b_amount_6": {
          "type": "double"
        },
        "performance_unit_7": {
          "type": "double"
        },
        "performance_a_amount_7": {
          "type": "double"
        },
        "performance_unit_8": {
          "type": "double"
        },
        "performance_a_amount_8": {
          "type": "double"
        },
        "performance_b_amount_8": {
          "type": "double"
        },
        "performance_unit_9": {
          "type": "double"
        },
        "performance_a_amount_9": {
          "type": "double"
        },
        "performance_b_amount_9": {
          "type": "double"
        },
        "performance_unit_10": {
          "type": "double"
        },
        "performance_a_amount_10": {
          "type": "double"
        },
        "performance_unit_11": {
          "type": "double"
        },
        "performance_a_amount_11": {
          "type": "double"
        },
        "performance_unit_12": {
          "type": "double"
        },
        "performance_a_amount_12": {
          "type": "double"
        },
        "performance_unit_13": {
          "type": "double"
        },
        "performance_a_amount_13": {
          "type": "double"
        },
        "performance_a_amount_14": {
          "type": "double"
        },
        "performance_a_amount_15": {
          "type": "double"
        },
        "performance_unit_15": {
          "type": "double"
        },
        "performance_a_amount_15": {
          "type": "double"
        }
      }
    }
  }
}
6

Below is the Aggregation that we use.

GET weekly_performance/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "week_end_date": {
            "format": "yyyy-MM-dd",
            "gte": "2017-01-28",
            "lte": "2017-02-04"
          }
          }
        },
        {
          "term": {
            "category1_id": 99
          }
        },
        {
          "term": {
            "category2_id": 9
          }
        }
      ]
    }
  },
  "aggs": {
    "group_by_performance_code": {
      "terms": {
        "size": 10000,
        "field": "performance_code"
      },
      "aggs": {
        "performance_unit_1": {
          "sum": {
            "field": "performance_unit_1"
          }
        },
        "performance_a_amount_1" :{
          "sum": {
            "field": "performance_a_amount_1"
          }
        },
        "performance_b_amount_1" :{
          "sum": {
            "field": "performance_b_amount_1"
          }
        },
        "performance_unit_2" :{
          "sum": {
            "field": "performance_unit_2"
          }
        },
        "performance_a_amount_2" :{
          "sum": {
            "field": "performance_a_amount_2"
          }
        },
        "performance_b_amount_2" :{
          "sum": {
            "field": "performance_b_amount_2"
          }
        },
        "performance_unit_3" :{
          "sum": {
            "field": "performance_unit_3"
          }
        },
        "performance_a_amount_3" :{
          "sum": {
            "field": "performance_a_amount_3"
          }
        },
        "performance_b_amount_3" :{
          "sum": {
            "field": "performance_b_amount_3"
          }
        },
        "performance_unit_4" :{
          "sum": {
            "field": "performance_unit_4"
          }
        },
        "performance_a_amount_4" :{
          "sum": {
            "field": "performance_a_amount_4"
          }
        },
        "performance_b_amount_4" :{
          "sum": {
            "field": "performance_b_amount_4"
          }
        },
        "performance_unit_5" :{
          "sum": {
            "field": "performance_unit_5"
          }
        },
        "performance_a_amount_5" :{
          "sum": {
            "field": "performance_a_amount_5"
          }
        },
        "performance_b_amount_5" :{
          "sum": {
            "field": "performance_b_amount_5"
          }
        },
        "performance_a_amount_6" :{
          "sum": {
            "field": "performance_a_amount_6"
          }
        },
        "performance_b_amount_6" :{
          "sum": {
            "field": "performance_b_amount_6"
          }
        },
        "performance_unit_7" :{
          "sum": {
            "field": "performance_unit_7"
          }
        },
        "performance_a_amount_7" :{
          "sum": {
            "field": "performance_a_amount_7"
          }
        },
        "performance_unit_8" :{
          "sum": {
            "field": "performance_unit_8"
          }
        },
        "performance_a_amount_8" :{
          "sum": {
            "field": "performance_a_amount_8"
          }
        },
        "performance_b_amount_8" :{
          "sum": {
            "field": "performance_b_amount_8"
          }
        },
        "performance_unit_9" :{
          "sum": {
            "field": "performance_unit_9"
          }
        },
        "performance_a_amount_9" :{
          "sum": {
            "field": "performance_a_amount_9"
          }
        },
        "performance_b_amount_9" :{
          "sum": {
            "field": "performance_b_amount_9"
          }
        },
        "performance_b_amount_15" :{
          "sum": {
            "field": "performance_b_amount_15"
          }
        },
        "performance_a_amount_15" :{
          "sum": {
            "field": "performance_a_amount_15"
          }
        },
        "performance_unit_15" :{
          "sum": {
            "field": "performance_unit_15"
          }
        },
        "agg_source": {
          "top_hits": {
            "size": 1,
            "_source": {
              "includes": ["category1_id", "category2_id", "category3_id", "category2_name", "category4_id", "category5_id","category6_id", "performance_code"]
            }
          }
        }
      }
        }
      }
    }

We are looking at getting the results < 1 second in most cases and <2 second in worst case scenario.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.