# Bulk inserting is slow **URL:** \ **Category:** Elasticsearch\ **Created:** [June 23, 2014, 10:30am UTC](https://discuss.elastic.co/t/bulk-inserting-is-slow/18259 "2014-06-23T10:30:14Z")\ **Posts on this page:** 1\ **Showing post:** 8
**Author:** ![phoenix](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phoenix/32/1319_2.png) [@phoenix](https://discuss.elastic.co/u/phoenix)\ **Post date:** [June 24, 2014, 3:59pm UTC](https://discuss.elastic.co/t/bulk-inserting-is-slow/18259/8 "2014-06-24T15:59:41Z")
Thanks to both of you, i'll look at this immediately ! Le mardi 24 juin 2014 17:51:04 UTC+2, Jörg Prante a écrit : > You should use the org.elasticsearch.action.bulk.BulkProcessor helper > class for concurrent bulk indexing. > > Jörg > > On Tue, Jun 24, 2014 at 5:34 PM, Frederic Esnault \<[esnault....@gmail.com](mailto:esnault....@gmail.com) > \\> wrote: > > > Hi again, > > > > any idea about how to parallelize the bulk insert process ? > > I tried creating 4 BulkInserters extending RecursiveAction and executed > > them all, but the result is awful, 3 of them finished very slowly, and one > > did not finish (don't know why), and got only 70K docs in ES instead of 265 > > 000... > > > > The result of downsizing the batches sizes to 10 000 is not really big, > > total process took approx. 1 second less (Actually this is much lower than > > in the previous post, because i moved the importing UI to my server, close > > to one of ES nodes). Was more than 29 seconds, now 28. > > 28 seconds. > > > > _Import CSV file took 28.069 secondes_ > > > > Here is the insertion code. The Iterator is a CSV reading iterator who > > parses lines and returns Record instances (object with generic object > > values, indexed as string). MAX\_RECORDS is my batch size, set to 10 000. > > > > ``` > > public void insert(Iterator recordsIterator) { > > while (recordsIterator.hasNext()) { > > batchInsert(recordsIterator, MAX_RECORDS); > > } > > } > > > > private void batchInsert(Iterator recordsIterator, int limit) > > > > ``` > > > > { > > BulkRequestBuilder bulkRequest = client.prepareBulk(); > > int processed = 0; > > try { > > logger.log(Level.INFO, "Adding records to bulk insert batch"); > > while (recordsIterator.hasNext() && processed \< limit) { > > processed++; > > Record record = recordsIterator.next(); > > IndexRequestBuilder builder = > > client.prepareIndex(datasetName, RECORD); > > XContentBuilder data = jsonBuilder(); > > data.startObject(); > > for (ColumnMetadata column : > > dataset.getMetadata().getColumns()) { > > Object value = > > record.getCell(column.getName()).getValue(); > > if (value == null || (value instanceof String && > > value.equals("NULL"))) { > > value = null; > > } > > data.field(column.getNormalizedName(), value); > > } > > data.endObject(); > > builder.setSource(data); > > bulkRequest.add(builder); > > } > > logger.log(Level.INFO, "Added "+ > > bulkRequest.numberOfActions() +" records to bulk insert batch. Inserting > > batch..."); > > long current = System.currentTimeMillis(); > > BulkResponse bulkResponse = > > bulkRequest.setConsistencyLevel(WriteConsistencyLevel.ONE).execute().actionGet(); > > if (bulkResponse.hasFailures()) { > > logger.log(Level.SEVERE, "Could not index : " + > > bulkResponse.buildFailureMessage()); > > } > > System.out > > .println(String.format("Bulk insert took %s > > secondes", NumberUtils > > .formatSeconds(((double) > > (System.currentTimeMillis() - current)) / 1000.0))); > > } catch (Exception e) { > > e.printStackTrace(); > > } > > } > > > > Le mardi 24 juin 2014 13:44:03 UTC+2, Frederic Esnault a écrit : > > > > > Thanks for all this. > > > > > > I changed my conf, removed all the thread pool config, reduced refresh > > > time to 5s according to Michael advice, and limited my batch to 10 000. > > > I'll see how it works then i'll paralellize the bulk insert. > > > I'll tell you how it ends up. > > > > > > Thanks again ! > > > > > > Le lundi 23 juin 2014 12:56:14 UTC+2, Jörg Prante a écrit : > > > > > > > Your bulk insert size is too large. It makes no sense to insert 100.000 > > > > with one request. Use 1000-10000 instead. > > > > > > > > Also you should submit bulk requests in parallel and not sequential > > > > like you do. Sequential bulk is slow if client CPU/network is not saturated. > > > > > > > > Check if you have disabled the index refresh from 1 (1s) to -1 while > > > > bulk indexing is active. 30s makes not much sense if you can execute the > > > > bulk in this time. > > > > > > > > Do not limit indexing memory to 50%. > > > > > > > > It makes no sense to increase queue\_size for bulk thread pool to 1000. > > > > This means you want a single ES node should accept 1000 x 100000 = 100 000 > > > > 000 = 100m docs at once. This will simply exceeds all reasonable limits and > > > > bring the node down with an OOM (if you really have 100m docs). > > > > > > > > More advice is possible if you can show your client code how you push > > > > docs to ES. > > > > > > > > Jörg > > > > > > > > On Mon, Jun 23, 2014 at 12:30 PM, Frederic Esnault \< > > > > [esnault....@gmail.com](mailto:esnault....@gmail.com)\> wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > I'm inserting around 265 000 documents into an Elasticsearch cluster > > > > > composed of 3 nodes (real servers). > > > > > On two servers i give Elasticsearch 20g of heap, on third one which > > > > > has 64g ram, i set 30g of heap for Elasticsearch. > > > > > > > > > > I set Elasticsearch configuration to : > > > > > > > > > > - 3 shards (1 per server) > > > > > - 0 replicas > > > > > - discovery.zen.ping.multicast.enabled: false (and giving on each > > > > > node the unicast hostnames of the two other nodes); > > > > > - and this : > > > > > > > > > > indices.memory.index\_buffer\_size: 50% > > > > > index.refresh\_interval: 30s > > > > > threadpool: > > > > > index: > > > > > type: fixed > > > > > size: 30 > > > > > queue\_size: 1000 > > > > > bulk: > > > > > queue\_size: 1000 > > > > > bulk: > > > > > type: fixed > > > > > size: 30 > > > > > queue\_size: 1000 > > > > > search: > > > > > type: fixed > > > > > size: 100 > > > > > queue\_size: 200 > > > > > get: > > > > > type: fixed > > > > > size: 100 > > > > > queue\_size: 200 > > > > > > > > > > Indexing is done by groups of 100 000 docs, and here is my application > > > > > log : > > > > > INFO: Adding records to bulk insert batch > > > > > INFO: Added 100000 records to bulk insert batch. Inserting batch... > > > > > -- Bulk insert took 38.724 secondes > > > > > INFO: Adding records to bulk insert batch > > > > > INFO: Added 100000 records to bulk insert batch. Inserting batch... > > > > > -- Bulk insert took 31.134 secondes > > > > > INFO: Adding records to bulk insert batch > > > > > INFO: Added 64201 records to bulk insert batch. Inserting batch... > > > > > -- Bulk insert took 17.366 secondes > > > > > > > > > > --- Import CSV file took 108.905 secondes --- > > > > > > > > > > I'm wondering if this time is correct or not, or if there is something > > > > > i can do to improve performances ? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/3a38e79e-9afb-4146-a7e1-7984ec082e22% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/3a38e79e-9afb-4146-a7e1-7984ec082e22%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3a38e79e-9afb-4146-a7e1-7984ec082e22%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/32280fd5-5879-4424-882d-5b4e7674557a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/32280fd5-5879-4424-882d-5b4e7674557a%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/32280fd5-5879-4424-882d-5b4e7674557a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/32280fd5-5879-4424-882d-5b4e7674557a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9fc68d44-caf9-4315-b846-29ac5e1f8988%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9fc68d44-caf9-4315-b846-29ac5e1f8988%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). --- _[View the full topic](https://discuss.elastic.co/t/bulk-inserting-is-slow/18259)._