Elasticsearch - question about shards

(Thomas Rohlik) #1

Hi there!

We have 2 ES instances in our environment with 2 daily index and we already have more than 7k shards.
It is normal?
If not, what are you suggest?

epoch      timestamp cluster       status node.total node.data shards  pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1527602653 16:04:13  logs green           2         2   7760 3880    0    0        0             0                  -                100.0%

(David Pilato) #2

You probably have too many shards per node.

May I suggest you look at the following resources about sizing:


(Thomas Rohlik) #3

Thank you for your links.
I figured out that our shards size is very small:

GET _cat/shards/logs-2018.05.29
logs-2018.05.29 1 r STARTED 196755 176.9mb db1
logs-2018.05.29 1 p STARTED 196755 176.7mb db2
logs-2018.05.29 3 r STARTED 197144 176.9mb db1
logs-2018.05.29 3 p STARTED 197144 176.6mb db2
logs-2018.05.29 4 p STARTED 196554 176.2mb db1
logs-2018.05.29 4 r STARTED 196554 177.4mb db2
logs-2018.05.29 2 p STARTED 196979 177.8mb db1
logs-2018.05.29 2 r STARTED 196979 176.7mb db2
logs-2018.05.29 0 p STARTED 196328 176.1mb db1
logs-2018.05.29 0 r STARTED 196328 176.5mb db2

As I understand, it is common to have shard size around 30GB. Currently I have number_of_shards set to 5 and this is probably my problem. I will set this parameter to 2.

(David Pilato) #4

Even better, set it to 1 :slight_smile:

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.