Very Slow incrementing counter

Hello,
I have a counter which i need to increment :

curl -XPOST 'localhost:9200/something/ble/372204/_update' -d' { "script": { "inline": "ctx._source.counter++" } }'

This index has around ~4k elements but every counter increment takes minimum 160-300ms. Incrementing the same counter in postgres on same data also with index takes 0.01 ms. I understand that one is written in C and other in Java but both make reindex and elastic is not 10-20x slower it's 1600x times slower.

Elastic version 2.3.4
Config/init changes:

ES_HEAP_SIZE=3g
bootstrap.mlockall: true
LimitMEMLOCK=infinity

Stats are here:

http://pastebin.com/fW9nS0Wi

Incrementing the counter requires reindexing the whole document. In a
system like postgresql that just involves copying the row, probably to the
same block. Usually you don't even need to mess with indexes. Elasticsearch
doesn't work like that. It has to reanalyze all the fields and eventually
build a new segment, inverted index, doc values, and all.

Relational databases make tradeoffs to make updates faster. Elasticsearch
makes tradeoffs to make aggregations and full text search faster. Those
tradeoffs are baked in at a fairly deep level.

That said 160ms is quite a bit. You have lots of fields in that document?
Network between the shards slowish? I dunno, hard to say.

Hi Nik,
Thx for replying. I don't have a lof of fields i think around 35. It's not network, it's on one server and i am doing local requests, time is spent inside elasticsearch. Any way to check why it's so slow?

Can you try with an indexed script?

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-scripting.html#_indexed_scripts

Are you using SSD drives?

Hi it dosen't change anything. I am not using SSD.

Any way to profile elastic and see where it's spending it's time on this query ?

Using SSD could help a bit here. Also from 5.0 switching to Painless scripting language could help a bit.

You could run hot_threads API while the query is running and see what is happening.

I don't remember exactly the code but here are the main steps elasticsearch has to run when you call this script:

  • compile the script (might be cached IIRC)
  • load the document
  • execute the script
  • delete the old version of the document
  • put the new version of the document (which leads to all what @nik9000 described)