# Poor ingest node and indexing performance

**URL:** <https://discuss.elastic.co/t/poor-ingest-node-and-indexing-performance/77962>\
**Category:** Elasticsearch\
**Created:** [March 9, 2017, 10:59am UTC](https://discuss.elastic.co/t/poor-ingest-node-and-indexing-performance/77962 "2017-03-09T10:59:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jsyu32](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jsyu32](https://discuss.elastic.co/u/jsyu32)\
**Post date:** [March 9, 2017, 10:59am UTC](https://discuss.elastic.co/t/poor-ingest-node-and-indexing-performance/77962/1 "2017-03-09T10:59:12Z")

</div>

We are using ElasticSearch 5.1.2:  
Bootstrap is passed except system\_call\_filter, because we are running on CentOS 6

Here is the setting:  
The document to be indexed have 1200 fields, each document is near 4K. But all the fields are consisted of three types: date, keyword, numberic. 24 millions records in total.  
We want to use ElasticSearch to do near real-time aggregations.  
We have three common nodes(master:true; data:true; ingest:default true) & one standalone ingest node(master:false; data:false; ingest:true). All these nodes jvm config are same as the following.Each node have 64GB memory, SSD, 24cores.

jvm.options:  
-Xms24g  
-Xmx24g  
-XX:NewSize=8g  
-XX:MaxNewSize=8g  
-XX:+UseConcMarkSweepGC  
-XX:CMSInitiatingOccupancyFraction=75  
-XX:+UseCMSInitiatingOccupancyOnly

We load data into ES by using elasticsearch-spark, DataFrame.saveToEs() method, which is bulk insert. We use 10 executers which means 10 clients to parallel insert to the ingest node.  
Here is the performance we got.  
Ingest Node's

![](https://us1.discourse-cdn.com/elastic/original/3X/6/5/6559d3cf9afb2c06ea5bf450ecb70938ba6f2227.png)  
detail  
 ![](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24b242941fb6f702dce9587cbe3bd69c3a34c4cd.png)

One of data node's

 ![](https://us1.discourse-cdn.com/elastic/original/3X/1/5/15de28bf2131b6bb39f034d965968d717c0a4fe2.png)

With 4 nodes, we only got index rate near 1000 document/s。 near 4 ms for indexing one document.  
All the nodes' cpu usage is under 10%. it won't be bottleneck.Don't have throttle, bulk queue, bulk rejections, index queue, index rejections or other issues.

Increase client, the performance will deteriorate.

Settings of Index:  
"settings": {  
"index": {  
"number\_of\_shards": 3,  
"number\_of\_replicas": 0,  
"mapping.total\_fields.limit": 1500,  
"refresh\_interval": "60s"  
}

Settings of Cluster  
Cluster:  
{  
"persistent": {  
"indices": {  
"store": {  
"throttle": {  
"max\_bytes\_per\_sec": "100mb"  
}  
}  
}  
},  
"transient": {}  
}

Other settings are default.

My question is why indexing performance is so poor, only 1000 document/s and why ingest node is so heavy, happened so much young gc, even full gc.  
From document, ingest does:  
• intercepts bulk and index requests  
• applies the transformations  
• passes the documents back to the index or bulk APIs  
We didn't use pipeline.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2017, 10:59am UTC](https://discuss.elastic.co/t/poor-ingest-node-and-indexing-performance/77962/2 "2017-04-06T10:59:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.