Dear ES forum users,
We have problem with importing documents to ElasticSearch in version
0.90.0, earlier we have been working on 0.20.2 and we did not experience
any problems (with adding documents to ES).
We are importing large documents, average size 5 MB (but there are much
larger to), number of documents 38 thousands, but inside of every document
is array with thousands of elements (that why they are so big).
We are using library version 0.90.0 (
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.0.zip→ elasticsearch-0.90.0\lib\elasticsearch-0.90.0.jar) in our importer. And
it is working properly with ES 0.20.1, but not so good with ES 0.90.0.
We are adding to ES 0.90.0 maximum amount of only 130 docs, with 2GB RAM
memory assigned to JVM. But with the same importer, but different
ElasticSearch service version (0.20.1) importer is importing all of 38ththousand documents without problem.
We think that ES (0.90.0) can not keep up with importing data.
Our logs says:
- Importer logs:
Exception in thread "calculations:AT-4831-01" org.elasticsearch.client.transport.NoNodeAvailableException: No node available
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:252)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:214)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:84)
at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java:310)
at org.elasticsearch.action.index.IndexRequestBuilder.doExecute(IndexRequestBuilder.java:315)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:62)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:57)
at com.pureholidayhomes.lib.es.ESConnector.send(ESConnector.java:30)
at com.pureholidayhomes.lib.es.Calculations.run(Calculations.java:398)
- ElasticSearch service log:
[2013-05-23 10:54:08,376][INFO ][node ] [Jumbo Carnation] {0.90.0}[4540]: initializing ...
[2013-05-23 10:54:08,376][INFO ][plugins ] [Jumbo Carnation] loaded [], sites []
[2013-05-23 10:54:11,246][INFO ][node ] [Jumbo Carnation] {0.90.0}[4540]: initialized
[2013-05-23 10:54:11,246][INFO ][node ] [Jumbo Carnation] {0.90.0}[4540]: starting ...
[2013-05-23 10:54:11,434][INFO ][transport ] [Jumbo Carnation] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {
inet[/192.168.1.15:9301]}
[2013-05-23 10:54:14,756][INFO ][cluster.service ] [Jumbo Carnation] detected_master [Hank Pym][wjgfN8BYQnaSMZpcIH_sFg][inet[/192.1
68.1.15:9300]], added {[Hank Pym][wjgfN8BYQnaSMZpcIH_sFg][inet[/192.168.1.15:9300]],}, reason: zen-disco-receive(from master [[Hank Pym][wjg
fN8BYQnaSMZpcIH_sFg][inet[/192.168.1.15:9300]]])
[2013-05-23 10:54:14,803][INFO ][discovery ] [Jumbo Carnation] elasticsearch/6g12eLWAT_qXtGvdYU_EHQ
[2013-05-23 10:54:14,897][INFO ][http ] [Jumbo Carnation] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {
inet[/192.168.1.15:9201]}
[2013-05-23 10:54:14,897][INFO ][node ] [Jumbo Carnation] {0.90.0}[4540]: started
[2013-05-23 11:01:37,788][WARN ][monitor.jvm ] [Jumbo Carnation] [gc][ParNew][433][122] duration [1.2s], collections [1]/[6.5s]
, total [1.2s]/[7.5s], memory [1.3gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.1mb]->[4.1mb]/[48mb]}{[Par Eden Space] [85.6mb]->[10.8mb]
/[532.5mb]}{[Par Survivor Space] [66.5mb]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[82mb]}
[2013-05-23 11:01:43,841][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][434][19] duration [5.6s], collection
s [1]/[6s], total [5.6s]/[13s], memory [1.2gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.1mb]->[4.1mb]/[48mb]}{[Par Eden Space] [10.8mb]-[70.9kb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.3gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[82
mb]}
[2013-05-23 11:01:49,956][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][435][20] duration [5.7s], collection
s [1]/[6.1s], total [5.7s]/[18.7s], memory [1.3gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.1mb]->[4.2mb]/[48mb]}{[Par Eden Space] [70.9
kb]->[4.1mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.3gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/
[82mb]}
[2013-05-23 11:01:55,728][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][436][21] duration [5.6s], collection
s [1]/[5.7s], total [5.6s]/[24.3s], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [4.1m
b]->[6.4mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[
82mb]}
[2013-05-23 11:02:01,438][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][437][22] duration [5.5s], collection
s [1]/[5.7s], total [5.5s]/[29.9s], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [6.4m
b]->[7.9mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[
82mb]}
[2013-05-23 11:02:49,611][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][476][28] duration [5.7s], collection
s [1]/[5.9s], total [5.7s]/[40.9s], memory [1.3gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [5.1m
b]->[7.1mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.3gb]->[1.3gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[
82mb]}
[2013-05-23 11:02:55,461][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][477][29] duration [5.6s], collection
s [1]/[5.8s], total [5.6s]/[46.6s], memory [1.3gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [7.1m
b]->[8.3mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.3gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[
82mb]}
[2013-05-23 11:03:01,295][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][478][30] duration [5.6s], collection
s [1]/[5.8s], total [5.6s]/[52.3s], memory [1.3gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [8.3m
b]->[9.1mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[
82mb]}
[2013-05-23 11:03:06,973][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][479][31] duration [5.4s], collection
s [1]/[5.6s], total [5.4s]/[57.8s], memory [1.3gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [9.1m
b]->[9.7mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[
82mb]}
[2013-05-23 11:03:13,239][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][481][32] duration [5s], collections
[1]/[5.2s], total [5s]/[1m], memory [1.3gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [93.4mb]->[3
.7mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[82mb]}
[2013-05-23 11:03:19,480][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][483][33] duration [5s], collections
[1]/[5.2s], total [5s]/[1.1m], memory [1.6gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.2mb]/[48mb]}{[Par Eden Space] [404.7mb]-[2.7mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[82m
b]}
[2013-05-23 11:03:25,860][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][484][34] duration [5.4s], collection
s [1]/[6.3s], total [5.4s]/[1.2m], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.3mb]/[48mb]}{[Par Eden Space] [2.7mb
]->[6.1mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]/[8
2mb]}
[2013-05-23 11:03:37,747][WARN ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][485][35] duration [10.7s], collectio
ns [1]/[11.8s], total [10.7s]/[1.4m], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.3mb]->[4.2mb]/[48mb]}{[Par Eden Space] [6.
1mb]->[5.4mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.6mb]
/[82mb]}
[2013-05-23 11:03:49,572][WARN ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][486][36] duration [10.7s], collectio
ns [1]/[11.8s], total [10.7s]/[1.5m], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.2mb]->[4.3mb]/[48mb]}{[Par Eden Space] [5.
4mb]->[10.2mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.6mb]->[31.7mb
]/[82mb]}
[2013-05-23 11:04:00,430][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][487][37] duration [9.8s], collection
s [1]/[10.8s], total [9.8s]/[1.7m], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.3mb]->[4.3mb]/[48mb]}{[Par Eden Space] [10.2
mb]->[10.3mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]
/[82mb]}
[2013-05-23 11:04:11,256][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][488][38] duration [9.8s], collection
s [1]/[10.8s], total [9.8s]/[1.9m], memory [1.2gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.3mb]->[4.3mb]/[48mb]}{[Par Eden Space] [10.3
mb]->[10.5mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]
/[82mb]}
[2013-05-23 11:04:23,409][WARN ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][490][39] duration [10.9s], collectio
ns [1]/[11s], total [10.9s]/[2m], memory [1.7gb]->[1.2gb]/[1.9gb], all_pools {[Code Cache] [4.3mb]->[4.3mb]/[48mb]}{[Par Eden Space] [485.4m
b]->[1.9mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[
82mb]}
[2013-05-23 11:04:35,124][WARN ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][491][40] duration [11.1s], collectio
ns [1]/[11.7s], total [11.1s]/[2.2m], memory [1.2gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.3mb]->[4.3mb]/[48mb]}{[Par Eden Space] [1.
9mb]->[10mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/
[82mb]}
[2013-05-23 11:04:47,025][WARN ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][492][41] duration [10.9s], collectio
ns [1]/[11.9s], total [10.9s]/[2.4m], memory [1.3gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.3mb]->[4.3mb]/[48mb]}{[Par Eden Space] [10
mb]->[11mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.2gb]->[1.2gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[
82mb]}
[2013-05-23 11:05:55,215][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][550][59] duration [5.7s], collection
s [1]/[6s], total [5.7s]/[2.6m], memory [1.3gb]->[1.3gb]/[1.9gb], all_pools {[Code Cache] [4.4mb]->[4.4mb]/[48mb]}{[Par Eden Space] [20.1mb]
->[19.2mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.3gb]->[1.3gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[8
2mb]}
[2013-05-23 11:05:55,231][INFO ][node ] [Jumbo Carnation] {0.90.0}[4540]: stopping ...
[2013-05-23 11:06:01,018][INFO ][monitor.jvm ] [Jumbo Carnation] [gc][ConcurrentMarkSweep][551][60] duration [5.2s], collection
s [1]/[5.8s], total [5.2s]/[2.7m], memory [1.3gb]->[1gb]/[1.9gb], all_pools {[Code Cache] [4.4mb]->[4.4mb]/[48mb]}{[Par Eden Space] [19.2mb]
->[7mb]/[532.5mb]}{[Par Survivor Space] [0b]->[0b]/[66.5mb]}{[CMS Old Gen] [1.3gb]->[1gb]/[1.3gb]}{[CMS Perm Gen] [31.7mb]->[31.7mb]/[82mb]}
Our guess is that ES is not releasing resources that should be released,
or computation for arrays is changed and now ES can not keep up with it, or
less memory is assigned to data handling in ES 0.90.0.
Please help us to resolve that problem.
Best Regards,
Gzom.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.