Hello
I use Java API to index 5 following documents into an index called javabulkindex and a mapping called javabulkmappingtype, using Java Bulk API with ES 0.18.4 and O.17.6:
BulkRequestBuilder BRB=client.prepareBulk();
BRB.add(client.prepareIndex().setIndex("javabulkindex").setType("javabulkmappingtype").setId("ID1").setSource("field", JSONObject.fromObject(JsonString1)));
BRB.add(client.prepareIndex().setIndex("javabulkindex").setType("javabulkmappingtype").setId("ID2").setSource("field", JSONObject.fromObject(JsonString2)));
BRB.add(client.prepareIndex().setIndex("javabulkindex").setType("javabulkmappingtype").setId("ID3").setSource("field", JSONObject.fromObject(JsonString2)));
BRB.add(client.prepareIndex().setIndex("javabulkindex").setType("javabulkmappingtype").setId("ID4").setSource("field", "value1"));
BRB.add(client.prepareIndex().setIndex("javabulkindex").setType("javabulkmappingtype").setId("ID5").setSource("field", "value2"));
BulkResponse br = BRB.execute().actionGet();
if (br.hasFailures()) {
System.out.println("\nbulk failure with error: " + br.buildFailureMessage());
}
T
-
if this index and mapping ( javabulkindex and javabulkmappingtype) have been created before (with number_of_shards=7, numbers_replicas=2), then all above commands have fails with error message "cannot allocate shards..."
-
if I delete this index and mapping and run these commands again, then:
- an index and mapping with these names are created
-all first three commands are successful, but the two last commands fail with error message "try to parse string value as object but it seems that they are provided a value" and " a problem with EOF...". Attention: these last two commands I used String instead of an object for field's value.
-if I delete first three commands, then these last two commands are successful.
So, please explain to me the reason and how to solve it.
Thanks a lot
Tran CD
Ngu ngoc lon nhat cua doi nguoi la yeu duong - dieu ran thu 15 cua duc Phat (tai sao khong)
TINH YEU KHONG TU NHIEN SINH RA, KHONG TU NHIEN MAT DI, NO CHI CHUYEN TU NGUOI NAY SANG NGUOI KHAC
NGAY MAI EM DI
BIEN NAM CHAN EM GIAT VE
De : Shay Banon kimchy@gmail.com
À : elasticsearch@googlegroups.com
Envoyé le : Dimanche 20 Novembre 2011 9h31
Objet : Re: java.lang.OutOfMemoryError - how to anticipate memory usage?
On Fri, Nov 18, 2011 at 3:57 PM, Michal Wegorek wegorekm@gmail.com wrote:
Problem:
Indexing tens of millions of small (200B) docs. During indexing
process memory is gradually rising, see:
http://i.imgur.com/xfI98.png
As soon as ES reaches max memory limit I receive WARN:
failed engine java.lang.OutOfMemoryError: Java heap space
Since that moment, node is dead.
Questions:
1)Is there any way to anticipate how much memory is needed if I want
to store 5,000,000,000 docs?
Hard to tell without understanding the structure of the docs, what get indexed, and so on. You can extrapolate on the memory that will be needed by indexing a subset of the docs.
2)Why memory is gradually rising, is the memory usage rising together
with number of docs?
Yes, the more docs you have the more memory will be needed, but its not linear. Also, memory usage relates to if faceting are used, sorting, number of fields indexed, how the text get analyzed.
My config:
Using 1 index, default min/max memory limits.
cluster.name: mwegorekTest
gateway.type: fs
gateway.fs.location: /mnt/mwegorek
gateway.fs.snapshot_interval: 30s
gateway.recover_after_nodes: 2
gateway.recover_after_time: 1m
gateway.expected_nodes: 4
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["test-indexer-1.atm:9300", "test-
indexer-2.atm:9300", "test-indexer-3.atm:9300", "test-indexer-4.atm:
9300"]
threadpool:
index:
type: blocking
min: 4
size: 10
wait_time: 60s
