Elasticsearch upsert is happening at slow rate

I am doing upsert to elasticsearch-5.0.2 in bulk having batch size of 5000.
When I start this upsert, ES response time is normal. But after few seconds, ES response time increases 20-30times. Logs for this:

Processing file :  1_1536077223904.log
ES_Response: 1724ms|Bucket Length: 5000
ES_Response: 535ms|Bucket Length: 5000
ES_Response: 686ms|Bucket Length: 5000
ES_Response: 614ms|Bucket Length: 5000
ES_Response: 505ms|Bucket Length: 5000
ES_Response: 5985ms|Bucket Length: 5000
ES_Response: 26915ms|Bucket Length: 5000
ES_Response: 15685ms|Bucket Length: 5000
ES_Response: 13839ms|Bucket Length: 5000
ES_Response: 14022ms|Bucket Length: 5000
ES_Response: 13766ms|Bucket Length: 5000
ES_Response: 13252ms|Bucket Length: 5000

So, it is taking on average 14seconds to upsert a batch of 5000 into the elasticsearch.

I am using elasticsearch client in nodeJS for this. I have also tried to run two instances of my program to upsert data. ES_reponse time doubles.
I have also turned off replica and increased refresh interval to 10sec but no luck.
Logs of ES is also normal.

ElasticSearch I am running in cluster (having 2 nodes deployed on different servers).
Specification of each server where ES is deployed:

Cores: 4
Total RAM: 16GB
Allocated RAM to ES: 4GB

Elasticsearch config i am using for queue:

thread_pool.search.queue_size: 500
thread_pool.bulk.queue_size: 500
thread_pool.index.queue_size: 500

bulk.size, index.size and search.size i am using default configuration.

Is it due to low hardware configuration or do i have change/add anything in elasticsearch config file?

Are you frequently updating the same documents? What does your data look like, e.g. are you using nested documents?

I am updating a given document three times.
Data is a flat json.
Below is the data sample:

{ "script" : { "inline": "if (ctx._source.mst == null) {ctx._source.mst = params.mst} if (ctx._source.sts == null) {ctx._source.sts = params.sts}", "lang" : "painless", "params" : {"mst": 0,"sts" : 0}}, "upsert" : {"mst" : 0, "sts": 0}}

I don't have much data in elasticsearch too.

Is there anything in the Elasticsearch logs that indicate a potential problem? Do you have monitoring installed?

Logs looks clean. no error/warning. Even GC is not running which can slow ingestion. I don't have monitoring installed. Shall I install it and monitor the performance?

Yes, installing monitoring can make it a lot easier to spot patterns.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.