Slow indexing rate and indexing time on index with constant document updates

Elastic Stack Elasticsearch
1

Hi,

We recently moved our backend to elasticsearch. As a part of the solution, we have a index which would require the constant updates to same document - on an average every 5 seconds.

We are doing this on our hot-path solution. But, constantly updating the same document underneath this index - is slowing us down (in terms of overall per document processing timings) on hot-path.

Details:

  • Using Elastic 2.2.1
  • Solution uses SSD on data nodes. There are total 6 data nodes - 8 core, 56 GB
  • Particular index has more than 150 properties. In terms of DTO, it has many sub-classes and sub-classes has sub-classes.
  • We have made all string type to not_analyzed
  • Index has 2 replicas.
  • Refresh interval is set to 30s
  • No compression on index
  • _all is disabled
  • Solution is C# based and we are using NEST for all communication
  • NEST only opens one HTTP connection with keep-alive set 2000 ms.
  • We never search on this particular index.
  • Always get the document directly using _id

On an average we see 800 ms to 4 seconds(in peak hours) time while indexing document in this particular index. We were hoping to get it down to under 100 ms - we tried couple of solution, but nothing seems to be working - in terms of improving end-to-end indexing time.

For testing and to see if we can get any better indexing timings, we changed following as well.

PUT /_cluster/settings
{
    "persistent" : {
        "indices.store.throttle.max_bytes_per_sec" : "200mb"
    }
}

PUT /_cluster/settings
{
    "transient" : {
        "indices.store.throttle.type" : "none" 
    }
}

However, the indexing time still remains the same.
In addition to this particular index, cluster also has two other rolling indices (new one gets created everyday). Both to these rolling indices has on an average 1 million documents getting indexed every 24 hours.

We are seeing longer indexing times on these two rolling indices as well - around 600 ms to 2.5 seconds.

Any suggestions would be really helpful for improving our indexing rate - which will allow us to process single document/message on hot-path in minimal amount of time.