# Poor performance indexing documents **URL:** \ **Category:** Elasticsearch\ **Created:** [October 28, 2013, 11:04pm UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147 "2013-10-28T23:04:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Mark\_Woon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_woon/32/2010_2.png) [@Mark\_Woon](https://discuss.elastic.co/u/Mark_Woon)\ **Post date:** [October 28, 2013, 11:04pm UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147/1 "2013-10-28T23:04:51Z")
I've installed ES 0.90.5 on a RHEL 6 server using the RPM. Using the default config, I start indexing my data. After about 200 documents or so, I'm getting 503's every 5 documents. The documents aren't large (e.g. [https://gist.github.com/markwoon/7206263](https://gist.github.com/markwoon/7206263)). It goes away after a couple seconds, but I consistently get a 503 after 5 more documents. I'm new to ES, so any pointers would be helpful on how I can improve indexing speed. I've looked at [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#setup-configuration-memory](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#setup-configuration-memory) I've changed ES\_HEAP\_SIZE to 8GB. I've checked that max\_file\_descriptors is up to 64K. I've tried setting bootstrap.mlockall to true. None has made much of a difference. I'm not even sure what kind of logging or testing I can do to figure out what my problem is. Any ideas? Thanks, -Mark -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Carofe](https://avatars.discourse-cdn.com/v4/letter/c/8baadc/32.png) [@Carofe](https://discuss.elastic.co/u/Carofe)\ **Post date:** [October 28, 2013, 11:13pm UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147/2 "2013-10-28T23:13:07Z")
I was able to increase the the indexing speed significantly doing the following (this is not exactly good for real-time indexing/searching requirements): before the indexing: number\_of\_shards:3 (something more than 1, it depends on your needs and resources) number\_of\_replicas: 0 refresh\_interval:-1 merge.policy.max\_merged\_segment: 1gb Do indexing after the indexing: set number\_of\_replicas: 1 (or whatever is good for your system) refresh manually with the API On Monday, October 28, 2013 4:04:51 PM UTC-7, Mark Woon wrote: > I've installed ES 0.90.5 on a RHEL 6 server using the RPM. Using the > default config, I start indexing my data. After about 200 documents or so, > I'm getting 503's every 5 documents. The documents aren't large (e.g. > [Sample JSON for MEDLINE data (PMID:12336567). · GitHub](https://gist.github.com/markwoon/7206263)). It goes away after a couple > seconds, but I consistently get a 503 after 5 more documents. > > I'm new to ES, so any pointers would be helpful on how I can improve > indexing speed. > > I've looked at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#setup-configuration-memory) > > I've changed ES\_HEAP\_SIZE to 8GB. > I've checked that max\_file\_descriptors is up to 64K. > I've tried setting bootstrap.mlockall to true. > > None has made much of a difference. > > I'm not even sure what kind of logging or testing I can do to figure out > what my problem is. Any ideas? > > Thanks, > -Mark -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 29, 2013, 9:07am UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147/3 "2013-10-29T09:07:09Z")
How do you index the data: What language is the client? Which API do you use? Can you show a stack trace of the 503 error, in your client, and maybe messages in the cluster log file? You do not need 8 GB heap, 64k file descriptors, or mlockall to index such a tiny amount of 200 docs. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Carofe](https://avatars.discourse-cdn.com/v4/letter/c/8baadc/32.png) [@Carofe](https://discuss.elastic.co/u/Carofe)\ **Post date:** [October 29, 2013, 4:07pm UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147/4 "2013-10-29T16:07:40Z")
On Tuesday, October 29, 2013 2:07:09 AM UTC-7, Jörg Prante wrote: > How do you index the data: What language is the client? Which API do you > use? Java API for everything. > Can you show a stack trace of the 503 error, in your client, and maybe > messages in the cluster log file? I don't actually have that error but my indexing was very slow (many millions of documents to index) > You do not need 8 GB heap, 64k file descriptors, or mlockall to index such > a tiny amount of 200 docs. I'm using 4gb per data node. That's more than enough for me. I think 1gb may be enough for you. > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mark\_Woon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_woon/32/2010_2.png) [@Mark\_Woon](https://discuss.elastic.co/u/Mark_Woon)\ **Post date:** [October 30, 2013, 1:01am UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147/5 "2013-10-30T01:01:20Z")
Solved my problem, and it had nothing to do with ES. There was a proxy in front of the ES server that was rate limiting it... -Mark On Monday, October 28, 2013 4:04:51 PM UTC-7, Mark Woon wrote: > I've installed ES 0.90.5 on a RHEL 6 server using the RPM. Using the > default config, I start indexing my data. After about 200 documents or so, > I'm getting 503's every 5 documents. The documents aren't large (e.g. > [Sample JSON for MEDLINE data (PMID:12336567). · GitHub](https://gist.github.com/markwoon/7206263)). It goes away after a couple > seconds, but I consistently get a 503 after 5 more documents. > > I'm new to ES, so any pointers would be helpful on how I can improve > indexing speed. > > I've looked at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#setup-configuration-memory) > > I've changed ES\_HEAP\_SIZE to 8GB. > I've checked that max\_file\_descriptors is up to 64K. > I've tried setting bootstrap.mlockall to true. > > None has made much of a difference. > > I'm not even sure what kind of logging or testing I can do to figure out > what my problem is. Any ideas? > > Thanks, > -Mark -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:09am UTC](https://discuss.elastic.co/t/poor-performance-indexing-documents/14147/6 "2017-07-06T02:09:55Z")