What is indexing latency?

Elastic Stack Elasticsearch
1

Hi, can someone explain a bit on what "indexing latency" is? If the indexing rate didn't change, but indexing latency going up, what does this mean? And it's also observed whenever indexing latency goes up, server CPU goes up as well. Does this mean I need to add in more elasticsearch node to handing the indexing? Thanks a lot!

2

Indexing latency is the time taken by the elastic node for indexing the document.
It will be impacted by the memory in your jvm and overall load on the Disk.
In case it has gone up , kindly check if load on your cluster. Increase in search load will impact the indexing too.
A single bad query can hamper the elastic performance.
Other pointers to improve performance are : -

  1. prefer bulk indexing

  2. ensure that load is evenly distributed on your cluster.
    etc.
    following link talks in length about what all you should do before you launch your cluster in production
    https://www.elastic.co/guide/en/elasticsearch/guide/current/_revisit_this_list_before_production.html

3

thanks a lot for the reply.
JVM are all in healthy range, and no changes on the amount of data ingested. I've already followed the document to make sure the right configurations. The only thing I can see is the increase of search rate. Since there are no way I can limit how users can search, will it help if I add in additional data node? And are there any ways we can see the searches from users? thanks.

4

If yours is self hosted solution you can add data node and to improve performance you can ensure that not more than 2-3 shards per core on a single node. The lesser the better.

Following are the most common things which people overlook up when they start using elastic

  • Hope your mapping is good and you don't have lots of text fields.
  • _all is disabled , you are using byte/short in place of int and long wherever applicable etc.
  • You have assigned not more than 30.5 GB as heap space to JVM and atleast 50% of the space is available on the node to be used by system. What i mean is if your host has X GB don't assign more than X/2 GB to elasticSearch process.
5

I don't have any index having more than 3 shards per data node.

  • I do have a lot of string fields, but most are defined as not analysed, except one or two, which must be analyzed to enable user for free text search.
  • In terms of int/long, this is probably something I've missed out.
  • Heap/memory assignment is good.
    Adding additional data node did help a bit, at least CPU doesn't spike as high as before. But I still don't understand what caused data node CPU spike high? Is more users searching concurrently a contributing factor? Anything else? Data ingestion to the cluster has been stable, and I've already cleared any errors in elasticsearch/logstash indexers/kibana log files.
6

Even we are facing simillar issue. CPU Spiking after cluster is up for around 20 hours and then to control it we have to restart the whole cluster.
This issue is being faced by many see https://discuss.elastic.co/t/cluster-a3900c-suddenly-slow/78283

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.