Setting total shard per node


(Wijaya) #1

Hi,

i have 30 data node,3 master node and 1 coordinator node, i want to create 1 index with 30k shard (each node gets 1k shard).
i try put mapping shard on coordinator node, response from node is error "Failed to parse value [30000] for setting [index.number_of_shards] must be <= 1024",
so please give me advices for this problem


(Mark Walkom) #2

Why would you want it that big?


(Wijaya) #3

i will put the data 2 PB, please give me advice for node topology how many data node, master node and coordinator node?


(Mark Walkom) #4

You would be much better off having multiple indices.


(Wijaya) #5

how to combine/append indices? to get all the data when I query


(Mark Walkom) #6

What sort of use case is this? What is the data source, what sort of queries are you doing?


(Wijaya) #7

the data source from log, sum of the data 1k per second, my query sql like this (select * from index union all select * from index2), sorry i am newbie in elasticsearch


(Mark Walkom) #8

You should use time based indices then, with one index per hour/day/week. not one big index.


(Wijaya) #9

how to retrieve all data from indices?


(Wijaya) #10

Thank Mark, i can get retrieve all data use GET /index*/_search


(David Pilato) #11

Yes you can.

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right


(Christian Dahlqvist) #12

Once you have gone through the resources David linked to, you may also find this webinar on storage optimisation useful.


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.