# Adding millions of documents, performance decay

**URL:** <https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902>\
**Category:** Elasticsearch\
**Created:** [November 30, 2012, 10:57am UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902 "2012-11-30T10:57:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fabio\_Pezzoni](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@Fabio\_Pezzoni](https://discuss.elastic.co/u/Fabio_Pezzoni)\
**Post date:** [November 30, 2012, 10:57am UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/1 "2012-11-30T10:57:29Z")

</div>

Adding millions of documents, performance decay.

Hi,

I'm new to ElasticSearch and I'm trying to transfer a database of several  
millions of JSON documents to a Lucene index through ES. Currently we can  
use just a single node with 8 CPUs and we use the Java API to add  
sequentially each document. We didn't changed the default options,  
therefore our index has 5 shards. At the beginning the process was very  
fast! In a few our we added about 50 millions of documents, then the  
performance gradually fell and currently it can take seconds to add a  
single document. Query performance are still very good!  
There is a way to overtake this situation? Maybe changing the setting or  
the number of shards...

Thank you very much, Fabio.

Here some node stats:

{"ok":true,"cluster\_name":"twitter","nodes":{"j5RTlp7jTreoAlE6Tb3gwA":{"name":"xxx","transport\_address":"inet[/xxx.xxx.xxx.xxx:9300]","hostname":"xxx","http\_address":"inet[/xxx.xxx.xxx.xxx:9200]","settings":{"path.home":"/home/twitter/elasticsearch-0.19.11","foreground":"yes","logger.prefix":"","max-open-files":"true","node.name":"xxx","cluster.name":"twitter","name":"xxx","path.logs":"/home/twitter/elasticsearch-0.19.11/logs"},"os":{"refresh\_interval":1000,"cpu":{"vendor":"Intel","model":"Xeon","mhz":3192,"total\_cores":8,"total\_sockets":8,"cores\_per\_socket":16,"cache\_size":"8kb","cache\_size\_in\_bytes":8192},"mem":{"total":"15.4gb","total\_in\_bytes":16543477760},"swap":{"total":"3.9gb","total\_in\_bytes":4294963200}},"process":{"refresh\_interval":1000,"id":20301,"max\_file\_descriptors":65535},"jvm":{"pid":20301,"version":"1.6.0\_34","vm\_name":"Java  
HotSpot(TM) 64-Bit Server VM","vm\_version":"20.9-b04","vm\_vendor":"Sun  
Microsystems  
Inc.","start\_time":1354178513836,"mem":{"heap\_init":"256mb","heap\_init\_in\_bytes":268435456,"heap\_max":"1011.2mb","heap\_max\_in\_bytes":1060372480,"non\_heap\_init":"23.1mb","non\_heap\_init\_in\_bytes":24313856,"non\_heap\_max":"130mb","non\_heap\_max\_in\_bytes":136314880,"direct\_max":"1011.2mb","direct\_max\_in\_bytes":1060372480}}}}}

Index stats:

{"ok":true,"\_shards":{"total":10,"successful":5,"failed":0},"\_all":{"primaries":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}},"total":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}},"indices":{"twitter":{"primaries":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}},"total":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}}}}}}

--

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [November 30, 2012, 4:38pm UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/2 "2012-11-30T16:38:45Z")

</div>

If you haven't done this yet, set refresh interval to -1 by running:

curl -XPUT localhost:9200/twitter/\_settings -d '{  
"index" : {  
"refresh\_interval" : "-1"  
}  
}'

when you are done with bulk reindexing you can turn it back on by running

curl -XPUT localhost:9200/twitter/\_settings -d '{  
"index" : {  
"refresh\_interval" : "1s"  
}  
}'

On Friday, November 30, 2012 5:57:29 AM UTC-5, Fabio Pezzoni wrote:

> Adding millions of documents, performance decay.
> 
> Hi,
> 
> I'm new to Elasticsearch and I'm trying to transfer a database of several  
> millions of JSON documents to a Lucene index through ES. Currently we can  
> use just a single node with 8 CPUs and we use the Java API to add  
> sequentially each document. We didn't changed the default options,  
> therefore our index has 5 shards. At the beginning the process was very  
> fast! In a few our we added about 50 millions of documents, then the  
> performance gradually fell and currently it can take seconds to add a  
> single document. Query performance are still very good!  
> There is a way to overtake this situation? Maybe changing the setting or  
> the number of shards...
> 
> Thank you very much, Fabio.
> 
> Here some node stats:
> 
> {"ok":true,"cluster\_name":"twitter","nodes":{"j5RTlp7jTreoAlE6Tb3gwA":{"name":"xxx","transport\_address":"inet[/xxx.xxx.xxx.xxx:9300]","hostname":"xxx","http\_address":"inet[/xxx.xxx.xxx.xxx:9200]","settings":{"path.home":"/home/twitter/elasticsearch-0.19.11","foreground":"yes","logger.prefix":"","max-open-files":"true","  
> node.name":"xxx","cluster.name":"twitter","name":"xxx","path.logs":"/home/twitter/elasticsearch-0.19.11/logs"},"os":{"refresh\_interval":1000,"cpu":{"vendor":"Intel","model":"Xeon","mhz":3192,"total\_cores":8,"total\_sockets":8,"cores\_per\_socket":16,"cache\_size":"8kb","cache\_size\_in\_bytes":8192},"mem":{"total":"15.4gb","total\_in\_bytes":16543477760},"swap":{"total":"3.9gb","total\_in\_bytes":4294963200}},"process":{"refresh\_interval":1000,"id":20301,"max\_file\_descriptors":65535},"jvm":{"pid":20301,"version":"1.6.0\_34","vm\_name":"Java  
> HotSpot(TM) 64-Bit Server VM","vm\_version":"20.9-b04","vm\_vendor":"Sun  
> Microsystems  
> Inc.","start\_time":1354178513836,"mem":{"heap\_init":"256mb","heap\_init\_in\_bytes":268435456,"heap\_max":"1011.2mb","heap\_max\_in\_bytes":1060372480,"non\_heap\_init":"23.1mb","non\_heap\_init\_in\_bytes":24313856,"non\_heap\_max":"130mb","non\_heap\_max\_in\_bytes":136314880,"direct\_max":"1011.2mb","direct\_max\_in\_bytes":1060372480}}}}}
> 
> Index stats:
> 
> {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0},"\_all":{"primaries":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}},"total":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}},"indices":{"twitter":{"primaries":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}},"total":{"docs":{"count":54298508,"deleted":24537},"store":{"size":"50.6gb","size\_in\_bytes":54435026631,"throttle\_time":"0s","throttle\_time\_in\_millis":0},"indexing":{"index\_total":63351487,"index\_time":"2d","index\_time\_in\_millis":173455844,"index\_current":0,"delete\_total":0,"delete\_time":"0s","delete\_time\_in\_millis":0,"delete\_current":0},"get":{"total":0,"time":"0s","time\_in\_millis":0,"exists\_total":0,"exists\_time":"0s","exists\_time\_in\_millis":0,"missing\_total":0,"missing\_time":"0s","missing\_time\_in\_millis":0,"current":0},"search":{"query\_total":185,"query\_time":"1.2m","query\_time\_in\_millis":73521,"query\_current":0,"fetch\_total":71,"fetch\_time":"4.6m","fetch\_time\_in\_millis":279532,"fetch\_current":0},"merges":{"current":6,"current\_docs":60,"current\_size":"462.5kb","current\_size\_in\_bytes":473615,"total":761368,"total\_time":"3.1d","total\_time\_in\_millis":276398054,"total\_docs":224483235,"total\_size":"251.7gb","total\_size\_in\_bytes":270307183589},"refresh":{"total":177694,"total\_time":"1.3d","total\_time\_in\_millis":116764453},"flush":{"total":7797,"total\_time":"1.2d","total\_time\_in\_millis":111453937}}}}}}

--

---

<div class="post-metadata">

**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\
**Post date:** [November 30, 2012, 5:33pm UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/3 "2012-11-30T17:33:20Z")

</div>

Also, have you tried to group you documents into bulk statements?

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

On Fri, Nov 30, 2012 at 11:38 AM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote:

> If you haven't done this yet, set refresh interval to -1 by running:
> 
> curl -XPUT localhost:9200/twitter/\_settings -d '{  
> "index" : {  
> "refresh\_interval" : "-1"  
> }  
> }'
> 
> when you are done with bulk reindexing you can turn it back on by running
> 
> curl -XPUT localhost:9200/twitter/\_settings -d '{  
> "index" : {  
> "refresh\_interval" : "1s"  
> }  
> }'
> 
> On Friday, November 30, 2012 5:57:29 AM UTC-5, Fabio Pezzoni wrote:
> 
> > Adding millions of documents, performance decay.
> > 
> > Hi,
> > 
> > I'm new to Elasticsearch and I'm trying to transfer a database of several  
> > millions of JSON documents to a Lucene index through ES. Currently we can  
> > use just a single node with 8 CPUs and we use the Java API to add  
> > sequentially each document. We didn't changed the default options,  
> > therefore our index has 5 shards. At the beginning the process was very  
> > fast! In a few our we added about 50 millions of documents, then the  
> > performance gradually fell and currently it can take seconds to add a  
> > single document. Query performance are still very good!  
> > There is a way to overtake this situation? Maybe changing the setting or  
> > the number of shards...
> > 
> > Thank you very much, Fabio.
> > 
> > Here some node stats:
> > 
> > {"ok":true,"cluster\_name" **:"twitter","nodes":{"**  
> > j5RTlp7jTreoAlE6Tb3gwA":{" **name":"xxx","transport\_**  
> > address":"inet[/xxx.xxx.xxx.**xxx:9300]","hostname":"xxx","**  
> > http\_address":"inet[/xxx.xxx.**xxx.xxx:9200]","settings":{"**  
> > path.home":"/home/twitter/ **elasticsearch-0.19.11","**  
> > foreground":"yes","logger.**prefix":"","max-open-files":"true","  
> > node.name":"xxx","cluster.name [http://cluster.name](http://cluster.name)  
> > ":"twitter","name":"xxx","path.logs":"/home/  
> > twitter/elasticsearch-0.19.11/logs"},"os":{"refresh\_  
> > interval":1000,"cpu":{"vendor"**:"Intel","model":"Xeon","mhz":\*\*  
> > 3192,"total\_cores":8,"total\_ **sockets":8,"cores\_per\_socket":**  
> > 16,"cache\_size":"8kb","cache\_ **size\_in\_bytes":8192},"mem":{"**  
> > total":"15.4gb","total\_in\_ **bytes":16543477760},"swap":{"**  
> > total":"3.9gb","total\_in\_ **bytes":4294963200}},"process":**  
> > {"refresh\_interval":1000,"id": **20301,"max\_file\_descriptors":**  
> > 65535},"jvm":{"pid":20301," **version":"1.6.0\_34","vm\_name":**"Java  
> > HotSpot(TM) 64-Bit Server VM","vm\_version":"20.9-b04","\*\*vm\_vendor":"Sun  
> > Microsystems Inc.","start\_time": **1354178513836,"mem":{"heap\_**  
> > init":"256mb","heap\_init\_in\_ **bytes":268435456,"heap\_max":"**  
> > 1011.2mb","heap\_max\_in\_bytes": **1060372480,"non\_heap\_init":"**  
> > 23.1mb","non\_heap\_init\_in\_ **bytes":24313856,"non\_heap\_max"**  
> > :"130mb","non\_heap\_max\_in\_ **bytes":136314880,"direct\_max":**  
> > "1011.2mb","direct\_max\_in\_\*\*bytes":1060372480}}}}}
> > 
> > Index stats:
> > 
> > {"ok":true,"_shards":{" **total":10,"successful":5,"**  
> > failed":0},"all":{"primaries" **:{"docs":{"count":54298508,"**  
> > deleted":24537},"store":{" **size":"50.6gb","size\_in\_bytes"**  
> > :54435026631,"throttle\_time":" **0s","throttle\_time\_in\_millis":**  
> > 0},"indexing":{"index\_total": **63351487,"index\_time":"2d","**  
> > index\_time\_in\_millis": **173455844,"index\_current":0,"**  
> > delete\_total":0,"delete\_time": **"0s","delete\_time\_in\_millis":**  
> > 0,"delete\_current":0},"get":{"\*\*total":0,"time":"0s","time\_in\*\*  
> > millis":0,"exists\_total":0,"\*\*exists\_time":"0s","exists_\*\*  
> > time\_in\_millis":0,"missing\_ **total":0,"missing\_time":"0s","**  
> > missing\_time\_in\_millis":0," **current":0},"search":{"query\_**  
> > total":185,"query\_time":"1.2m" **,"query\_time\_in\_millis":73521,**  
> > "query\_current":0,"fetch\_ **total":71,"fetch\_time":"4.6m",**  
> > "fetch\_time\_in\_millis":279532, **"fetch\_current":0},"merges":{"**  
> > current":6,"current\_docs":60," **current\_size":"462.5kb","**  
> > current\_size\_in\_bytes":473615, **"total":761368,"total\_time":"**  
> > 3.1d","total\_time\_in\_millis": **276398054,"total\_docs":**  
> > 224483235,"total\_size":"251. **7gb","total\_size\_in\_bytes":**  
> > 270307183589},"refresh":{" **total":177694,"total\_time":"1.**  
> > 3d","total\_time\_in\_millis": **116764453},"flush":{"total":**  
> > 7797,"total\_time":"1.2d"," **total\_time\_in\_millis":**  
> > 111453937}},"total":{"docs":{" **count":54298508,"deleted":**  
> > 24537},"store":{"size":"50. **6gb","size\_in\_bytes":**  
> > 54435026631,"throttle\_time":" **0s","throttle\_time\_in\_millis":**  
> > 0},"indexing":{"index\_total": **63351487,"index\_time":"2d","**  
> > index\_time\_in\_millis": **173455844,"index\_current":0,"**  
> > delete\_total":0,"delete\_time": **"0s","delete\_time\_in\_millis":**  
> > 0,"delete\_current":0},"get":{" **total":0,"time":"0s","time\_in\_**  
> > millis":0,"exists\_total":0," **exists\_time":"0s","exists\_**  
> > time\_in\_millis":0,"missing\_ **total":0,"missing\_time":"0s","**  
> > missing\_time\_in\_millis":0," **current":0},"search":{"query\_**  
> > total":185,"query\_time":"1.2m" **,"query\_time\_in\_millis":73521,**  
> > "query\_current":0,"fetch\_ **total":71,"fetch\_time":"4.6m",**  
> > "fetch\_time\_in\_millis":279532, **"fetch\_current":0},"merges":{"**  
> > current":6,"current\_docs":60," **current\_size":"462.5kb","**  
> > current\_size\_in\_bytes":473615, **"total":761368,"total\_time":"**  
> > 3.1d","total\_time\_in\_millis": **276398054,"total\_docs":**  
> > 224483235,"total\_size":"251. **7gb","total\_size\_in\_bytes":**  
> > 270307183589},"refresh":{" **total":177694,"total\_time":"1.**  
> > 3d","total\_time\_in\_millis": **116764453},"flush":{"total":**  
> > 7797,"total\_time":"1.2d"," **total\_time\_in\_millis":**  
> > 111453937}},"indices":{" **twitter":{"primaries":{"docs":**  
> > {"count":54298508,"deleted": **24537},"store":{"size":"50.**  
> > 6gb","size\_in\_bytes": **54435026631,"throttle\_time":"**  
> > 0s","throttle\_time\_in\_millis": **0},"indexing":{"index\_total":**  
> > 63351487,"index\_time":"2d"," **index\_time\_in\_millis":**  
> > 173455844,"index\_current":0," **delete\_total":0,"delete\_time":**  
> > "0s","delete\_time\_in\_millis": **0,"delete\_current":0},"get":{"**  
> > total":0,"time":"0s","time\_in\_ **millis":0,"exists\_total":0,"**  
> > exists\_time":"0s","exists\_ **time\_in\_millis":0,"missing\_**  
> > total":0,"missing\_time":"0s"," **missing\_time\_in\_millis":0,"**  
> > current":0},"search":{"query\_ **total":185,"query\_time":"1.2m"**  
> > ,"query\_time\_in\_millis":73521, **"query\_current":0,"fetch\_**  
> > total":71,"fetch\_time":"4.6m", **"fetch\_time\_in\_millis":279532,**  
> > "fetch\_current":0},"merges":{" **current":6,"current\_docs":60,"**  
> > current\_size":"462.5kb"," **current\_size\_in\_bytes":473615,**  
> > "total":761368,"total\_time":" **3.1d","total\_time\_in\_millis":**  
> > 276398054,"total\_docs": **224483235,"total\_size":"251.**  
> > 7gb","total\_size\_in\_bytes": **270307183589},"refresh":{"**  
> > total":177694,"total\_time":"1. **3d","total\_time\_in\_millis":**  
> > 116764453},"flush":{"total": **7797,"total\_time":"1.2d","**  
> > total\_time\_in\_millis": **111453937}},"total":{"docs":{"**  
> > count":54298508,"deleted": **24537},"store":{"size":"50.**  
> > 6gb","size\_in\_bytes": **54435026631,"throttle\_time":"**  
> > 0s","throttle\_time\_in\_millis": **0},"indexing":{"index\_total":**  
> > 63351487,"index\_time":"2d"," **index\_time\_in\_millis":**  
> > 173455844,"index\_current":0," **delete\_total":0,"delete\_time":**  
> > "0s","delete\_time\_in\_millis": **0,"delete\_current":0},"get":{"**  
> > total":0,"time":"0s","time\_in\_ **millis":0,"exists\_total":0,"**  
> > exists\_time":"0s","exists\_ **time\_in\_millis":0,"missing\_**  
> > total":0,"missing\_time":"0s"," **missing\_time\_in\_millis":0,"**  
> > current":0},"search":{"query\_ **total":185,"query\_time":"1.2m"**  
> > ,"query\_time\_in\_millis":73521, **"query\_current":0,"fetch\_**  
> > total":71,"fetch\_time":"4.6m", **"fetch\_time\_in\_millis":279532,**  
> > "fetch\_current":0},"merges":{" **current":6,"current\_docs":60,"**  
> > current\_size":"462.5kb"," **current\_size\_in\_bytes":473615,**  
> > "total":761368,"total\_time":" **3.1d","total\_time\_in\_millis":**  
> > 276398054,"total\_docs": **224483235,"total\_size":"251.**  
> > 7gb","total\_size\_in\_bytes": **270307183589},"refresh":{"**  
> > total":177694,"total\_time":"1. **3d","total\_time\_in\_millis":**  
> > 116764453},"flush":{"total": **7797,"total\_time":"1.2d","**  
> > total\_time\_in\_millis":\*\*111453937}}}}}}
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 30, 2012, 7:27pm UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/4 "2012-11-30T19:27:18Z")

</div>

You experience massive GC because you use the default out-of-the-box  
maximum JVM heap settings of 1 GB. You are lucky you could even add 50  
million docs with that small setting! But, you have 16 GB RAM. As a rule of  
thumb, assign around 50% RAM (4-8 GB) to Elasticsearch's heap. Check  
bin/elasticsearch.in.sh for ES\_MAX\_MEM or better ES\_HEAP\_SIZE. Other  
advanced tuning is also available, but, check bulk indexing first. Happy  
indexing!

Best regards,

Jörg

--

---

<div class="post-metadata">

**Author:** ![Fabio\_Pezzoni](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@Fabio\_Pezzoni](https://discuss.elastic.co/u/Fabio_Pezzoni)\
**Post date:** [December 3, 2012, 10:13am UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/5 "2012-12-03T10:13:39Z")

</div>

Thank you very much for the advises! I was already using Java API bulk  
statements. Now with 8g of heap and refresh\_interval=-1 it works far  
better. It's still slower than at the beginning but maybe it's normal  
for a single-node cluster (it indexes in bursts). I hope to have more  
nodes and power soon!

Fabio

On Fri, Nov 30, 2012 at 8:27 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:

> You experience massive GC because you use the default out-of-the-box maximum  
> JVM heap settings of 1 GB. You are lucky you could even add 50 million docs  
> with that small setting! But, you have 16 GB RAM. As a rule of thumb, assign  
> around 50% RAM (4-8 GB) to Elasticsearch's heap. Check  
> bin/elasticsearch.in.sh for ES\_MAX\_MEM or better ES\_HEAP\_SIZE. Other  
> advanced tuning is also available, but, check bulk indexing first. Happy  
> indexing!
> 
> Best regards,
> 
> Jörg
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Hadar\_Rottenberg](https://avatars.discourse-cdn.com/v4/letter/h/49beb7/32.png) [@Hadar\_Rottenberg](https://discuss.elastic.co/u/Hadar_Rottenberg)\
**Post date:** [December 4, 2012, 1:57pm UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/6 "2012-12-04T13:57:02Z")

</div>

Hey Fabbio,  
Would it be possible for you to post some benchmarks and statistics about  
your data set.  
such as  
how big is the dataset?  
what is the avg document size?  
how long did it take to index 50M documents?  
querying benchmarks?queries per second?

Thanks

On Monday, December 3, 2012 12:13:39 PM UTC+2, Fabio Pezzoni wrote:

> Thank you very much for the advises! I was already using Java API bulk  
> statements. Now with 8g of heap and refresh\_interval=-1 it works far  
> better. It's still slower than at the beginning but maybe it's normal  
> for a single-node cluster (it indexes in bursts). I hope to have more  
> nodes and power soon!
> 
> Fabio
> 
> On Fri, Nov 30, 2012 at 8:27 PM, Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com)\<javascript:\>\>  
> wrote:
> 
> > You experience massive GC because you use the default out-of-the-box  
> > maximum  
> > JVM heap settings of 1 GB. You are lucky you could even add 50 million  
> > docs  
> > with that small setting! But, you have 16 GB RAM. As a rule of thumb,  
> > assign  
> > around 50% RAM (4-8 GB) to Elasticsearch's heap. Check  
> > bin/elasticsearch.in.sh for ES\_MAX\_MEM or better ES\_HEAP\_SIZE. Other  
> > advanced tuning is also available, but, check bulk indexing first. Happy  
> > indexing!
> > 
> > Best regards,
> > 
> > Jörg
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/adding-millions-of-documents-performance-decay/9902/7 "2017-07-06T03:01:37Z")

</div>