Elasticsearch upsert is happening at slow rate

Elastic Stack Elasticsearch
1

I am doing upsert to elasticsearch-5.0.2 in bulk having batch size of 5000.
When I start this upsert, ES response time is normal. But after few seconds, ES response time increases 20-30times. Logs for this:

Processing file :  1_1536077223904.log
ES_Response: 1724ms|Bucket Length: 5000
ES_Response: 535ms|Bucket Length: 5000
ES_Response: 686ms|Bucket Length: 5000
ES_Response: 614ms|Bucket Length: 5000
ES_Response: 505ms|Bucket Length: 5000
ES_Response: 5985ms|Bucket Length: 5000
ES_Response: 26915ms|Bucket Length: 5000
ES_Response: 15685ms|Bucket Length: 5000
ES_Response: 13839ms|Bucket Length: 5000
ES_Response: 14022ms|Bucket Length: 5000
ES_Response: 13766ms|Bucket Length: 5000
ES_Response: 13252ms|Bucket Length: 5000

So, it is taking on average 14seconds to upsert a batch of 5000 into the elasticsearch.

I am using elasticsearch client in nodeJS for this. I have also tried to run two instances of my program to upsert data. ES_reponse time doubles.
I have also turned off replica and increased refresh interval to 10sec but no luck.
Logs of ES is also normal.

ElasticSearch I am running in cluster (having 2 nodes deployed on different servers).
Specification of each server where ES is deployed:

Cores: 4
Total RAM: 16GB
Allocated RAM to ES: 4GB

Elasticsearch config i am using for queue:

thread_pool.search.queue_size: 500
thread_pool.bulk.queue_size: 500
thread_pool.index.queue_size: 500

bulk.size, index.size and search.size i am using default configuration.

Is it due to low hardware configuration or do i have change/add anything in elasticsearch config file?

2

Are you frequently updating the same documents? What does your data look like, e.g. are you using nested documents?

3

I am updating a given document three times.
Data is a flat json.
Below is the data sample:

{"update":{"_index":"abc20180904","_type":"typename","_id":"111153608948_C125702","_retry_on_conflict":3}}
{"doc":{"fnlts":1536089956447,"jt":1,"id":"111153608948_C125702","pid":"1111536089482723_1","uid":19,"aid":8,"ct":"3.1","it":"5.5","snr":"919999999999","fsnr":"919999999999","rec":"918989","mc":"4.6","cc":"4","dndc":"Music","nec":0,"pri":2,"nt":0,"fc":1,"dre":"1","cmn":"Jobtype1","cht":"2.1","dur":444234,"msg":"hI","dcid":"1","pkgId":"1","st":1536089507000,"dl":"en","lg":"en","lc":"blah","recId":1,"prid":"0","jc":250000,"cid":"1111536089482723","cmt":1,"trc":500000},"doc_as_upsert":true}
{"update":{"_index":"abc20180904","_type":"typename","_id":"111153608948_C125702","_retry_on_conflict":3}}
{ "script" : { "inline": "if (ctx._source.mst == null) {ctx._source.mst = params.mst} if (ctx._source.sts == null) {ctx._source.sts = params.sts}", "lang" : "painless", "params" : {"mst": 0,"sts" : 0}}, "upsert" : {"mst" : 0, "sts": 0}}

I don't have much data in elasticsearch too.

4

Is there anything in the Elasticsearch logs that indicate a potential problem? Do you have monitoring installed?

5

Logs looks clean. no error/warning. Even GC is not running which can slow ingestion. I don't have monitoring installed. Shall I install it and monitor the performance?

6

Yes, installing monitoring can make it a lot easier to spot patterns.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.