Setting total shard per node

(Wijaya) #1


i have 30 data node,3 master node and 1 coordinator node, i want to create 1 index with 30k shard (each node gets 1k shard).
i try put mapping shard on coordinator node, response from node is error "Failed to parse value [30000] for setting [index.number_of_shards] must be <= 1024",
so please give me advices for this problem

(Mark Walkom) #2

Why would you want it that big?

(Wijaya) #3

i will put the data 2 PB, please give me advice for node topology how many data node, master node and coordinator node?

(Mark Walkom) #4

You would be much better off having multiple indices.

(Wijaya) #5

how to combine/append indices? to get all the data when I query

(Mark Walkom) #6

What sort of use case is this? What is the data source, what sort of queries are you doing?

(Wijaya) #7

the data source from log, sum of the data 1k per second, my query sql like this (select * from index union all select * from index2), sorry i am newbie in elasticsearch

(Mark Walkom) #8

You should use time based indices then, with one index per hour/day/week. not one big index.

(Wijaya) #9

how to retrieve all data from indices?

(Wijaya) #10

Thank Mark, i can get retrieve all data use GET /index*/_search

(David Pilato) #11

Yes you can.

May I suggest you look at the following resources about sizing:


(Christian Dahlqvist) #12

Once you have gone through the resources David linked to, you may also find this webinar on storage optimisation useful.

(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.