Hi, When indexing 200000 documents in batches of 1000 using the bulk REST API I am seeing the following log messages: [2016-11-03T21:30:55,263][INFO ][o.e.i.IndexingMemoryController] [node-1] now throttling indexing for shard [[myindex][1]]: segment writing can't keep up [2016-11-03T21:30:57,708…

So I modified the JVM options from: -Xms2g -Xmx2g To: -Xms20g -Xmx20g And the throttle message has stopped but I now get this instead: [2016-11-05T19:41:27,922][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][465] overhead, spent [443ms] collecting in the last [1s] [2016-11-05T19:41:2…

I thought it was perhaps excessive...oh well. I guess I will need to look for other reasons as to why the performance is 5 times slower than sphinx.

[image] trisport88: I guess I will need to look for other reasons as to why the performance is 5 times slower than sphinx. You haven't really given much to go on here, so it's very hard to say.

Well I'm not the expert... Maybe some pointers on how to increase the performance or things to look into? I freely admit it could be related to the adapter I wrote (although it is simply pulling information from SQL and pushing to Elastic using the bulk rest API. 1000 per second). Sphinx index tim…

What version are you on? What do the docs look like, what about the mapping?

What version are you on? 5.0 GA What do the docs look like, what about the mapping? Anonymized to protect the innocent... Mappings: {"index_one":{"mappings":{"document_one":{"_source":{"enabled":false},"properties":{"field1":{"type":"text"},"created_at":{"type":"long"},"field2":{"type":"…

[image] trisport88: I should add than my indexing program is single threaded. For optimal indexing performance you should probably increase this a bit. Don't start with a very large value, but rather gradually increase it in small steps until you see no further improvement in throughput. When…

Thanks Christian, I appreciate your suggestions (& link). I will investigate further.

This is excessive garbage collection, the node is spending more than 30% of its time collecting. That's why we have this warning.

Discuss the Elastic Stack

ES5 Indexing performance (seems slow)

Elastic Stack Elasticsearch

warkolm (Mark Walkom) November 6, 2016, 2:51am 3

It's garbage collection, it's normal.

Cross cluster search stops working after some time. ES: 6.1.1 Kibana: 6.0

Topic		Replies	Views
Improving Bulk Indexing Elasticsearch	12	4764	July 6, 2017
Indexing performance degrading over time Elasticsearch	67	33942	July 5, 2017
Rapidly Degrading Bulk Indexing Performance Elasticsearch	7	430	July 6, 2017
Issue Indexing 50mil Docs via Bulk API Elasticsearch	23	2527	July 5, 2017
Adding millions of documents, performance decay Elasticsearch	6	691	July 6, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

ES5 Indexing performance (seems slow)

Related topics