Couple of questions:
a, Can you provide more details on your bulk sizes? size, frequency,
synchronous/asynchronous, etc. I'm facing some challenges on the response
times from bulk indexing
https://discuss.elastic.co/t/bulk-import-response-times/27918.
Sorry I don't currently have solid numbers on response times in this area.
My indexing is the Reduce phase of a Map/Reduce job. So it is parallel,
but I limit the number of reducers to 1 or 2 per node in the cluster. I
was looking at the code today and it is currently configured at 100
messages per batch. It was a few years ago that I did substantial testing
of batch size and found the diminishing returns with larger batches.
b, m3.2xlarge is 8 core whereas m3.xlarge has 4. Do you think the number of
cores made the difference or was it convincing that the extra heap space is
the truly differentiating factor?
In my case I moved from c3.xlarge to m3.xlarge, so memory was the real
difference. The cluster size did not change. In your case I think you
achieve a similar result by reducing the number of nodes when you step up
to the m3.2xlarge or m4.2xlarge. This will keep your costs and cores
similar but double your RAM per node, improving efficiency of the RAM.
c, Were you able to figure out what goes into the 2GB of space? It would
appear that instance types like c3.large (4G memory) instances won't even
seem like a good choice for elasticsearch.
The 2GB is just a rule of thumb that I have used as a buffer. It is
loosely based on other best practices I have encountered over the years but
more so just on the reality that the OS needs some resources too, and if
you don't leave enough for the OS, then you will have issues. Also,
remember that anything the OS doesn't use directly it will use for file
caching which significantly benefits ElasticSearch as well, so you arent
really loosing anything by leaving this buffer. I have run ElasticSearch
on smaller instances, it works for small projects, with small
requirements. But yes I would strongly recommend against anything under
8GB, especially if you have more than 2 nodes. I.E. scale up a bit before
scaling out. A 16 node cluster of m3.mediums is nonsense. In my
consulting practice I strongly recommend targeting 64GB per node, and
scaling up instead of out until you reach that size. I really wish that
AWS wasn't so stingy with memory, but that is a common challenge in all
virtualized environments.