How to define shard count per index?

Elastic Stack Elasticsearch
1

Hi,

I don't know, if the post in ES is correct or if logstash forum fits better.

As I understand, the default chard size for indexes created by logstash is 5.
Currently we have 1 index per day for all types which have about 10Gb per day. 5 shards per index.

Now I would like to redesign a bit, to use multiple indexes. That would allow us to have different retention times (time until delete) per index which holds a set of logfiles / types.

We are currently using only a single instance of ELK.
To reduce the amount of memory needed by ES, I would like to decrease the shard size per index.

Here are my questions:

  1. Does the memory (RAM) consumed by shards only depend on the count of shards, or does it also depend on the size of the shards / indizes?
  2. How can I set the shard count on index level? Can it be done via logstash with some parameter, or do I need / modify the shard count in ES? How?
  3. Is my understanding correct, that I can have different shard count on different indizes? Can I change the chard count on the next index rotation without the need of reindexing old indizes with different shard size?

Thanks,
Andreas

2

How big is your heap size and what do yo want to decrease it too?

The number of shards while yes they have some memory over head is not that big. The use of the heap comes more in play on the amount of work you have to do. Searching, caching, indexing , and how many you can handle at the same time.

Are you using Java 1.8 , you might not know this but they have changed the HEAP usage model , They are now basically using Real memory and only allocate what they need (Even if your max heap is set) Basically PermGen has been removed and replace with "MetaSpace" so even if you have the Xmx set it still will only consume what it needs not the MAX of what is allocated like it used to

3

Sorry forgot to include what your asking for

Some good reading


https://www.elastic.co/guide/en/elasticsearch/reference/5.1/_basic_concepts.html

How an index gets created
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

4

thanks for the reply.

I created an Index within ES with the following statement:

[rootcurl -XPUT 'localhost:9200/metricbeat-2016.12.28?pretty' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1,
            "number_of_replicas" : 1
        }
    }
}'

so far, so good. But when Logstash is creating the next index for the next day, it sets shards again to 5. How can I tell logstash to keep the shard size set before or how can i set the shard size as parameter in logstash output?

5

I found it out.

If i am using index templates, it is working. Used this one:

curl -XPUT localhost:9200/_template/template_metricbeat -d '
{
    "template" : "metricbeat-*",
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "type1" : {
            "_source" : { "enabled" : false }
        }
    }
}
'
1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.