# How to define shard count per index?

**URL:** <https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787>\
**Category:** Elasticsearch\
**Created:** [January 6, 2017, 2:14pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787 "2017-01-06T14:14:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [January 6, 2017, 2:14pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787/1 "2017-01-06T14:14:14Z")

</div>

Hi,

I don't know, if the post in ES is correct or if logstash forum fits better.

As I understand, the default chard size for indexes created by logstash is 5.  
Currently we have 1 index per day for all types which have about 10Gb per day. 5 shards per index.

Now I would like to redesign a bit, to use multiple indexes. That would allow us to have different retention times (time until delete) per index which holds a set of logfiles / types.

We are currently using only a single instance of ELK.  
To reduce the amount of memory needed by ES, I would like to decrease the shard size per index.

Here are my questions:

1. Does the memory (RAM) consumed by shards only depend on the count of shards, or does it also depend on the size of the shards / indizes?
2. How can I set the shard count on index level? Can it be done via logstash with some parameter, or do I need / modify the shard count in ES? How?
3. Is my understanding correct, that I can have different shard count on different indizes? Can I change the chard count on the next index rotation without the need of reindexing old indizes with different shard size?

Thanks,  
Andreas

---

<div class="post-metadata">

**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\
**Post date:** [January 6, 2017, 2:48pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787/2 "2017-01-06T14:48:40Z")

</div>

How big is your heap size and what do yo want to decrease it too?

The number of shards while yes they have some memory over head is not that big. The use of the heap comes more in play on the amount of work you have to do. Searching, caching, indexing , and how many you can handle at the same time.

Are you using Java 1.8 , you might not know this but they have changed the HEAP usage model , They are now basically using Real memory and only allocate what they need (Even if your max heap is set) Basically PermGen has been removed and replace with "MetaSpace" so even if you have the Xmx set it still will only consume what it needs not the MAX of what is allocated like it used to

---

<div class="post-metadata">

**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\
**Post date:** [January 6, 2017, 2:56pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787/3 "2017-01-06T14:56:07Z")

</div>

Sorry forgot to include what your asking for

Some good reading

> **[Optimizing Elasticsearch: How Many Shards per Index?](https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index)**
>
> A key question in the minds of most Elasticsearch users when they create an index is “How many shards should I use?" In this article, we explain the design tradeoffs and performance consequences of choosing different values for the number of shards....

  
[https://www.elastic.co/guide/en/elasticsearch/reference/5.1/\_basic\_concepts.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/_basic_concepts.html)

How an index gets created  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html)

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [January 6, 2017, 4:02pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787/4 "2017-01-06T16:02:40Z")

</div>

thanks for the reply.

I created an Index within ES with the following statement:

```
[rootcurl -XPUT 'localhost:9200/metricbeat-2016.12.28?pretty' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1,
            "number_of_replicas" : 1
        }
    }
}'

```

so far, so good. But when Logstash is creating the next index for the next day, it sets shards again to 5. How can I tell logstash to keep the shard size set before or how can i set the shard size as parameter in logstash output?

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [January 6, 2017, 4:13pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787/5 "2017-01-06T16:13:03Z")

</div>

I found it out.

If i am using index templates, it is working. Used this one:

```
curl -XPUT localhost:9200/_template/template_metricbeat -d '
{
    "template" : "metricbeat-*",
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "type1" : {
            "_source" : { "enabled" : false }
        }
    }
}
'
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 3, 2017, 4:13pm UTC](https://discuss.elastic.co/t/how-to-define-shard-count-per-index/70787/6 "2017-02-03T16:13:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.