Index/Shard Design on ES cluster


(bob.webman) #1

Hi Guys,

I have 4 nodes in my ES cluster. They are big boxes with 24GB memory and
32TB hard disk. ES is configured with 12GB and I have done extensive
testing and I am happy with the actual implementation.

This ES cluster is connected to my Hadoop cluster with good 10GB
connections throughout. The hadoop cluster has 12 nodes and I use logstash
to move historical logs off the hadoop cluster to ES.

Given these assumptions:

  • I have lots of disk space per machine, I do not expect to run out of disk
    space.
  • The user query load is very light. Used for adhoc research not production
  • I will have several years of data so was planning on one index per month.
    eg; logstash-2014.90
  • I do not care too much about replication as all the data is on the hadoop
    cluster. On failure I will re-index

Question:

  • How many shards should I aim for per index?

I was thinking of FOUR per index on the assumption that it will be ONE
shard per node.
When I load the data from hadoop I do it via a streaming map-reduce using
the logstash netcat route with 3 hadoop nodes pointing to 1 ES node.
For this reason 1 shard per node seems a good idea?

eg;
hadoop1 streaming Mapper ----> logstash on hadoop ---->netcat-----> ES node1
hadoop2 streaming Mapper ----> logstash on hadoop ---->netcat
hadoop3 streaming Mapper ----> logstash on hadoop ---->netcat
hadoop4 streaming Mapper ----> logstash on hadoop ---->netcat-----> ES node2
hadoop5 streaming Mapper ----> logstash on hadoop ---->netcat
hadoop6 streaming Mapper ----> logstash on hadoop ---->netcat
...and so on

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/33b83ebd-b601-409c-91f6-42444586bcc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Your assumption is correct, one node per shard. You can over allocate if
you expect or want to add more nodes in the future.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bscRDxEdj-4Pbh_E983ZLMMjMpXj4irf%3D3T7zOjZw92w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3