Setting total shard per node

Elastic Stack Elasticsearch
1

Hi,

i have 30 data node,3 master node and 1 coordinator node, i want to create 1 index with 30k shard (each node gets 1k shard).
i try put mapping shard on coordinator node, response from node is error "Failed to parse value [30000] for setting [index.number_of_shards] must be <= 1024",
so please give me advices for this problem

2

Why would you want it that big?

3

i will put the data 2 PB, please give me advice for node topology how many data node, master node and coordinator node?

4

You would be much better off having multiple indices.

5

how to combine/append indices? to get all the data when I query

6

What sort of use case is this? What is the data source, what sort of queries are you doing?

7

the data source from log, sum of the data 1k per second, my query sql like this (select * from index union all select * from index2), sorry i am newbie in elasticsearch

8

You should use time based indices then, with one index per hour/day/week. not one big index.

9

how to retrieve all data from indices?

10

Thank Mark, i can get retrieve all data use GET /index*/_search

11

Yes you can.

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

12

Once you have gone through the resources David linked to, you may also find this webinar on storage optimisation useful.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.