Hi,
I'm new to ElasticSearch and I tried using Java API with bulk indexing.
I wrote simple Java program that tries to enter 1000000 documents.
However at the beginning I'm able to enter 1000 documents at less than 0.5
second while after some thousands of documents it takes more than 2 seconds
and keep increasing.
See output below and code as well.
I assume I'm doing something wrong. Please help me out. Thank you in
advance.
Regards,
Moshe
Output:
processed 1000 records from 5000 until 6000 at 781
processed 1000 records from 6000 until 7000 at 451
processed 1000 records from 7000 until 8000 at 683
processed 1000 records from 8000 until 9000 at 548
processed 1000 records from 9000 until 10000 at 303
processed 1000 records from 10000 until 11000 at 468
processed 1000 records from 11000 until 12000 at 316
processed 1000 records from 12000 until 13000 at 446
processed 1000 records from 13000 until 14000 at 342
processed 1000 records from 14000 until 15000 at 243
processed 1000 records from 15000 until 16000 at 252
processed 1000 records from 16000 until 17000 at 362
processed 1000 records from 17000 until 18000 at 302
processed 1000 records from 18000 until 19000 at 485
processed 1000 records from 19000 until 20000 at 402
processed 1000 records from 20000 until 21000 at 334
processed 1000 records from 21000 until 22000 at 429
processed 1000 records from 22000 until 23000 at 522
processed 1000 records from 23000 until 24000 at 434
processed 1000 records from 24000 until 25000 at 543
processed 1000 records from 25000 until 26000 at 476
processed 1000 records from 26000 until 27000 at 784
processed 1000 records from 27000 until 28000 at 797
processed 1000 records from 28000 until 29000 at 808
processed 1000 records from 29000 until 30000 at 670
processed 1000 records from 30000 until 31000 at 693
processed 1000 records from 31000 until 32000 at 710
processed 1000 records from 32000 until 33000 at 792
processed 1000 records from 33000 until 34000 at 582
processed 1000 records from 34000 until 35000 at 745
processed 1000 records from 35000 until 36000 at 762
processed 1000 records from 36000 until 37000 at 864
processed 1000 records from 37000 until 38000 at 880
processed 1000 records from 38000 until 39000 at 822
processed 1000 records from 39000 until 40000 at 1293
processed 1000 records from 40000 until 41000 at 1248
processed 1000 records from 41000 until 42000 at 1355
processed 1000 records from 42000 until 43000 at 999
processed 1000 records from 43000 until 44000 at 815
processed 1000 records from 44000 until 45000 at 934
processed 1000 records from 45000 until 46000 at 1213
processed 1000 records from 46000 until 47000 at 1085
processed 1000 records from 47000 until 48000 at 1136
processed 1000 records from 48000 until 49000 at 1352
processed 1000 records from 49000 until 50000 at 1417
processed 1000 records from 50000 until 51000 at 1423
processed 1000 records from 51000 until 52000 at 1275
processed 1000 records from 52000 until 53000 at 1215
processed 1000 records from 53000 until 54000 at 1324
processed 1000 records from 54000 until 55000 at 1132
processed 1000 records from 55000 until 56000 at 1268
processed 1000 records from 56000 until 57000 at 1055
processed 1000 records from 57000 until 58000 at 1618
processed 1000 records from 58000 until 59000 at 1608
processed 1000 records from 59000 until 60000 at 1600
processed 1000 records from 60000 until 61000 at 1673
processed 1000 records from 61000 until 62000 at 1340
processed 1000 records from 62000 until 63000 at 1815
processed 1000 records from 63000 until 64000 at 1708
processed 1000 records from 64000 until 65000 at 1543
processed 1000 records from 65000 until 66000 at 1674
processed 1000 records from 66000 until 67000 at 2005
processed 1000 records from 67000 until 68000 at 1889
processed 1000 records from 68000 until 69000 at 1570
processed 1000 records from 69000 until 70000 at 1527
processed 1000 records from 70000 until 71000 at 1603
processed 1000 records from 71000 until 72000 at 1748
processed 1000 records from 72000 until 73000 at 1745
processed 1000 records from 73000 until 74000 at 1580
processed 1000 records from 74000 until 75000 at 1588
processed 1000 records from 75000 until 76000 at 2291
processed 1000 records from 76000 until 77000 at 2903
processed 1000 records from 77000 until 78000 at 1625
processed 1000 records from 78000 until 79000 at 2087
Source code:
Node node = NodeBuilder.nodeBuilder().node();
Client client = node.client();
BulkRequestBuilder bulkRequest = client.prepareBulk();
int numOfDocs = 1000000;
long startTime = System.currentTimeMillis();
System.out.println("Going to add " + numOfDocs);
// either use client#prepare, or use Requests# to directly build
index/delete requests
long internalStartTime = System.currentTimeMillis();
for (int i = 0; i < numOfDocs; i++)
{
IndexRequestBuilder index =client.prepareIndex("twitter", "tweet", "id221"
+i);
index.setSource(jsonBuilder()
.startObject()
.field("user", "kimchy" +i)
.field("postDate", new Date())
.field("message", "trying out Elasticsearch"+i)
.endObject());
bulkRequest.add(index);
if (i % 1000 == 0 )
{
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures())
{
BulkItemResponse item[] = bulkResponse.getItems();
for (int j = 0; j< item.length; j++)
{
if (item[i].isFailed())
{
System.out.println ("Error " + item[i].getFailureMessage());
}
}
// bulkRequest = client.prepareBulk();
}
System.out.println ("processed 1000 records from " + (i-1000) + " until "
- i + " at " + (System.currentTimeMillis() - internalStartTime));
internalStartTime = System.currentTimeMillis();
}
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures())
{
BulkItemResponse item[] = bulkResponse.getItems();
for (int i = 0; i< item.length; i++)
{
if (item[i].isFailed())
{
System.out.println ("Error " + item[i].getFailureMessage());
}
}
}
System.out.println ("Finished entereing " + numOfDocs + " in " +
(System.currentTimeMillis() - startTime));
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0678d5a7-2034-4e99-bfe7-751d73af472a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
