Help Understanding Sharding

Elastic Stack Elasticsearch
1

Greetings All,

I'm a little fuzzy on the concept of sharding and have a few questions.

My ElasticSearch cluster consists of 4 Linux nodes. Each node has a 1 TB data disk. I'm shipping IIS logs to the cluster using Filebeat. I have the following environments...

NA1 - 12 web servers
NA2 - 9 web servers
NA3 - 12 web servers
NA4 - 9 web servers
STG - 4 web servers

When I started shipping logs I noticed that only two ElasticSearch nodes were getting data. There was only a single primary shard and single replica shard (on separate servers). I was under the impression that ElasticSearch would have evenly distributed the data across all nodes. I want to achieve the following.

Index Shards
na1_iis_logs split into 4 shards
na2_iis_logs split into 4 shards
na3_iis_logs split into 4 shards
na4_iis_logs split into 4 shards
stg_iis_logs split into 4 shards

How can I do this?

2

What version are you on?

3

if I am not mistaken then each indice will be by default in to 5 seperate shard.
and one replica.
and all goes evently in to different datanode.

you might have two node define as datanode.

here are some command to see what is happenning

Cluster Detail

curl -XGET hostname:9200/_cluster/health?pretty

Node status

curl -XGET hostname:9200/_nodes/stats?pretty

Shard Allocation detail

curl -XGET hostname:9200/_cluster/allocation/explain?pretty
curl -XGET hostname:9200/_cat/allocation?v

See name of all indice

curl -XGET hostname:9200/_cat/indices?pretty

4

This is true for versions prior to 7.0.0, but in 7.0.0 and later the default is 1 shard.

5

David,
I just upgraded my cluster to 7.0.1 today.
where is that setting then?

6

I don't really understand. Do you mean the number_of_shards setting? There's no setting for the default behaviour, because it's the default behaviour.

7

yes how to change that default setting. because I would like to keep five shard for all my indice

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.