i use bulkIndex or BulkProcessor Index Data ,and with 5 thread! my data are one milion items and 700M; this takes two hours to index! i think it's too long,how can better it ? how can i index faster?

In order for anyone to be able to help you, you will need to provide more details. How are you performing your bulk indexing? What is the size of your bulk requests? What is the size and structure of your documents? What does your cluster look like? What version of Elasticsearch are you using? How m…

sorry,! my elastticsearch version is 1.7. my cluster are two nodes,and i query data from orcale and foreach it this is my code： public static void ProceIndex(Client client, List plist){ BulkProcessor bulkProcessor = BulkProcessor.builder(client, new BulkProcessor.Listener() { long time1=Syste…

I would recommend lowering the size of the bulk requests. Large bulk requests does not necessarily result in improved performance. Try setting it to a few thousand documents and a maximum size of around 5MB. When you are indexing, what does the cluster look like? Are you saturating disk IO or possi…

Thanks! i tried to set replisas to 0 ,but is seems to not work really. my yml config like this: #index.analysis.analyzer.default.type : "ik" cluster.name: elasticsearch node.name: "node2" transport.tcp.port: 9302 "number_of_replicas": "0" "index.refresh_interval": "-1" [image]

Do not change the default values in the Elasticsearch config file to achieve this. Instead update the index settings through the API. Change replicas to 0 for the index prior to starting the bulk job and then set it back to the default value once the job has finished.

Thanks for your advise.i just update the bulkActions and BulkSize and number_of_replicas, now my indexing seems like this,this size in the head page is changing when i refesh the page,but the docs is 0.? is there any reason for it ?is it only show when the processing end ？ thanks!

and the logs in the console show that the work is indexing. before bulk----- 当前页:163 afterBulk bulk----- 总共索引:,耗时:0秒---- before bulk----- afterBulk bulk----- 总共索引:,耗时:0秒----

How is the cluster looking during indexing? Is there anything in the logs? How does CPU and disk IO look? If performance does not increase when setting replicas to 0 and there is no identifiable factor limiting performance, I would recommend performing a separate indexing benchmark with similar doc…

mybe in the for cycle,it query date from orcale DB,i select it every page and pagesize is 5000 items,my total datas is 26000000 items. select one page takes 100 seconds. do you have any idea for deal it ?

Discuss the Elastic Stack

How Can I increase ES's indexing Data speed?Bulk can't achieve it!

Elastic Stack Elasticsearch

LDA-MAN (LDA) December 15, 2015, 10:21am 9

Thanks!i update this ,but seems not better!

Topic		Replies	Views
How to increase indexing speed? Elasticsearch	5	5474	April 18, 2017
Indexing Performance, Threads + Bulk Size Elasticsearch	2	419	July 6, 2017
Bulk indexing slow down when data amount increase Elasticsearch	6	3012	July 6, 2017
BulkProcessor Indexing Performance Elasticsearch	1	1182	November 1, 2017
Slow bulk indexing Elasticsearch	4	2116	July 5, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

How Can I increase ES's indexing Data speed?Bulk can't achieve it!

Related topics