We might have faced similar problem with ES 1.3.6. The reason we found
was might be due to concurrent merges. These settings have helped us
in fixing the issue.
merge:
policy:
max_merge_at_once: 5
reclaim_deletes_weight: 4.0
segments_per_tier: 5
indices:
store:
throttle:
max_bytes_per_sec: 40mb # as we have few SATA disk for storage
type: merge
Ok a few things that don't make sense to me;
- 10 indexes of only ~220Kb? Are you sure of this? 2. If so why
not just one index? 3. Is baseball_data.json the data for an entire
index? If not can you clarify. 4. What java version are you on? 5.
What monitoring were you using? 6. Can you delete all your data,
switch monitoring on, start reindexing and then watch what happens?
Marvel would be ideal for this.
What you are seeing is really, really weird. That is a high shard
count however given the dataset is small I wouldn't think it'd
cause problems (but I could be wrong).
On 25 December 2014 at 02:27, Chris Moore <cmoore@perceivant.com
mailto:cmoore@perceivant.com> wrote:
Attached is the script we've been using to load the data and the
dataset. This is the mapping and a sample document
{ "baseball_1" : { "mappings" : { "team" : { "properties" : { "L" :
{ "type" : "integer", "store" : true }, "W" : { "type" :
"integer", "store" : true }, "name" : { "type" : "string", "store"
: true }, "teamID" : { "type" : "string", "store" : true },
"yearID" : { "type" : "string", "store" : true } } } } } }
{"yearID":"1871", "teamID":"PH1", "W":"21", "L":"7",
"name":"Philadelphia Athletics"}
On Wednesday, December 24, 2014 10:22:00 AM UTC-5, Chris Moore
wrote:
We tried many different test setups yesterday. The first setup we
tried was:
1 Master, 2 Data nodes 38 indices 10 shards per index 1 replica per
index 760 total shards (380 primary, 760 total) Each index had
2,745 documents Each index was 218.9kb in size (according to the
_cat/indices API)
We realize that 10 shards per index with only 2 nodes is not a good
idea, so we changed that and reran the tests.
We changed shards per index to the default of 5 and put 100 indices
on the 2 boxes and ran into the same issue. It was the same
dataset, so all other size information is correct.
After that, we turned off one of the data nodes, set replicas to 0
and shards per index to 1. With the same dataset, I loaded ~440
indices and ran into the timeout issues with the Master and Data
nodes just idling.
This is just a test dataset that we came up with to quickly test
our issues that contains no confidential information. Once we
figure out the issues affecting this test dataset, we'll try things
with our real dataset.
All of this works fine on ES 1.1.2, but not on 1.3.x (1.3.5 is our
current test version). We have also tried our real setup on 1.4.1
to no avail.
On Tuesday, December 23, 2014 5:03:30 PM UTC-5, Mark Walkom wrote:
Can you elaborate on your dataset and structure; how many indexes,
how many shards, how big they are etc.
On 24 December 2014 at 07:36, Chris Moore cmo...@perceivant.com
wrote:
Updating again:
If we reduce the number of shards per node to below ~350, the
system operates fine. Once we go above that (number_of_indices *
number_of_shards_per_index * number_of_replicas / number_of_nodes),
we start running into the described issues.
On Friday, December 12, 2014 2:11:08 PM UTC-5, Chris Moore wrote:
Just a quick update, we duplicated our test environment to see if
this issue was fixed by upgrading to 1.4.1 instead. We received the
same errors under 1.4.1.
On Friday, December 5, 2014 4:52:05 PM UTC-5, Chris Moore wrote:
As a followup, I closed all the indices on the cluster. I would
then open 1 index and optimize it down to 1 segment. I made it
through ~60% of the indices (and probably ~45% of the data) before
the same errors showed up in the master log and the same behavior
resumed.
On Friday, December 5, 2014 3:57:12 PM UTC-5, Chris Moore wrote:
I replied once, but it seems to have disappeared, so if this gets
double posted, I'm sorry.
We disabled all monitoring when we started looking into the issues
to ensure there was no external load on ES. Everything we are
currently seeing is just whatever activity ES generates
internally.
My understanding regarding optimizing indices is that you shouldn't
call it explicitly on indices that are regularly updating, rather
you should let the background merge process handle things. As the
majority of our indices regularly update, we don't explicitly call
optimize on them. I can try to call it on them all and see if it
helps.
As for disk speed, we are currently running ES on SSDs. We have it
in our roadmap to change that to RAIDed SSDs, but it hasn't been a
priority as we have been getting acceptable performance thus far.
On Friday, December 5, 2014 2:59:11 PM UTC-5, Jörg Prante wrote:
Do you have a monitor tool running?
I recommend to switch it off, and optimize your indices, and then
update your monitoring tools.
Seems you have many segments/slow disk to get them reported in
15s.
Jörg
Am 05.12.2014 16:10 schrieb "Chris Moore" cmo...@perceivant.com:
This is running on Amazon EC2 in a VPC on dedicated instances.
Physical network infrastructure is likely fine. Are there specific
network issues you think we should look into?
When we are in a problem state, we can communicate between the
nodes just fine. I can run curl requests to ES (health checks, etc)
from the master node to the data nodes directly and they return as
expected. So, there doesn't seem to be a socket exhaustion issue
(additionally there are no kernel errors being reported).
It feels like there is a queue/buffer filling up somewhere that
once it has availability again, things start working. But,
/_cat/thread_pool?v doesn't show anything above 0 (although, when
we are in the problem state, it doesn't return a response if run on
master), nodes/hot_threads doesn't show anything going on, etc.
On Thursday, December 4, 2014 4:10:37 PM UTC-5, Support Monkey
wrote:
I would think the network is a prime suspect then, as there is no
significant difference between 1.2.x and 1.3.x in relation to
memory usage. And you'd certainly see OOMs in node logs if it was a
memory issue.
On Thursday, December 4, 2014 12:45:58 PM UTC-8, Chris Moore
wrote:
There is nothing (literally) in the log of either data node after
the node joined events and nothing in the master log between index
recovery and the first error message.
There are 0 queries run before the errors start occurring (access
to the nodes is blocked via a firewall, so the only communications
are between the nodes). We have 50% of the RAM allocated to the
heap on each node (4GB each).
This cluster operated without issue under 1.1.2. Did something
change between 1.1.2 and 1.3.5 that drastically increased idle heap
requirements?
On Thursday, December 4, 2014 3:29:23 PM UTC-5, Support Monkey
wrote:
Generally __ReceiveTimeoutTransp____ortExcepti__on is due to
network disconnects or a node failing to respond due to heavy load.
What does the log of pYi3z5PgRh6msJX_armz_A show you? Perhaps it
has too little heap allocated. Rule of thumb is 1/2 available
memory but <= 31GB
On Wednesday, December 3, 2014 12:52:58 PM UTC-8, Jeff Keller
wrote:
ES Version: 1.3.5
OS: Ubuntu 14.04.1 LTS
Machine: 2 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 8 GB RAM at
AWS
master (ip-10-0-1-18), 2 data nodes (ip-10-0-1-19, ip-10-0-1-20)
After upgrading from ES 1.1.2...
- Startup ES on master 2. All nodes join cluster 3. [2014-12-03
20:30:54,789][INFO ][gateway ] [ip-10-0-1-18.ec2.internal]
recovered [157] indices into cluster_state 4. Checked health a few
times
curl -XGET localhost:9200/_cat/health?v
- 6 minutes after cluster recovery initiates (and 5:20 after the
recovery finishes), the log on the master node (10.0.1.18)
reports:
[2014-12-03
20:36:57,532][DEBUG][action.__ad____min.cluster.node.stats]
[ip-10-0-1-18.ec2.internal] failed to execute on node
[pYi3z5PgRh6msJX_armz_A]
org.elasticsearch.transport.Re____ceiveTimeoutTransportExcepti__on__:
[ip-10-0-1-20.ec2.internal][__in____et[/10.0.1.20:9300]][__cluster/__n__odes/stats/n]