When indexing a few hours duration, a timeout error occurs

Elastic Stack Elasticsearch
1

Hello,
I have a program, continuing to submit bulk to Elasticsearch, each bulk has
5000 documents, the number of concurrent requests is 8, such lasted several
hours later, the program appeared timeout error:

2014-09-12 09:23:29,092 INFO [org.elasticsearch.client.transport] - <[James
Jaspers] failed to get node info for
[#transport#-1][node1][inet[/192.60.9.4:9300]], disconnecting...>
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[/192.60.9.4:9300]][cluster/nodes/info] request_id [202756] timed
out after [5000ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
~[elasticsearch-1.1.2.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[na:1.6.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[na:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]

My elasticsearch.yml reads as follows:
index.merge.policy.max_merged_segment: 1gb
index.merge.policy.segments_per_tier: 4
index.merge.policy.max_merge_at_once: 4
index.merge.policy.max_merge_at_once_explicit: 4
index.merge.scheduler.max_thread_count: 1
indices.memory.index_buffer_size: 33%
indices.store.throttle.type: none
threadpool.merge.type: fixed
threadpool.merge.size: 4
threadpool.merge.queue_size: 32
threadpool.bulk.type: fixed
threadpool.bulk.size: 12
threadpool.bulk.queue_size: 32
bootstrap.mlockall: true
node.max_local_storage_nodes: 2

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5f72db0e-01fe-4e9e-afdb-9440baf51645%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

And I noticed RES and SHR growing, is this normal?

https://lh4.googleusercontent.com/-Jd-AWZSDemA/VBJVOQBo-iI/AAAAAAAAAQY/CYofZYrHeOc/s1600/QQ截图20140912093559.jpg

https://lh6.googleusercontent.com/-25qwZeO2_TA/VBJUkFPLg4I/AAAAAAAAAQQ/tk4Lx7EntCQ/s1600/QQ截图20140912093616.jpg
BTW, my ES_MIN_MEM = ES_MAX_MEM = 8g , a four-node cluster on two machines.

在 2014年9月12日星期五UTC+8上午10时00分37秒，Sephen Xu写道：

Hello,
I have a program, continuing to submit bulk to Elasticsearch, each bulk
has 5000 documents, the number of concurrent requests is 8, such lasted
several hours later, the program appeared timeout error:

2014-09-12 09:23:29,092 INFO [org.elasticsearch.client.transport] -
<[James Jaspers] failed to get node info for
[#transport#-1][node1][inet[/192.60.9.4:9300]], disconnecting...>
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[inet[/192.60.9.4:9300]][cluster/nodes/info] request_id [202756] timed
out after [5000ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
~[elasticsearch-1.1.2.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[na:1.6.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[na:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]

My elasticsearch.yml reads as follows:
index.merge.policy.max_merged_segment: 1gb
index.merge.policy.segments_per_tier: 4
index.merge.policy.max_merge_at_once: 4
index.merge.policy.max_merge_at_once_explicit: 4
index.merge.scheduler.max_thread_count: 1
indices.memory.index_buffer_size: 33%
indices.store.throttle.type: none
threadpool.merge.type: fixed
threadpool.merge.size: 4
threadpool.merge.queue_size: 32
threadpool.bulk.type: fixed
threadpool.bulk.size: 12
threadpool.bulk.queue_size: 32
bootstrap.mlockall: true
node.max_local_storage_nodes: 2

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/41e91518-5789-4782-81bf-d22d68045a28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.