Fail to update index data on Elasticsearch

Hi team,

I am facing an issue while using an Elasticsearch update query. I have given the detailed description of the issue below:

Version of Elasticsearch: 7.17.0

Problem Description:

My application has different teams. For each team, there is some data that is stored on elasticsearch with an index name format as given below:


Eg. attributes-2eXX7a00-XXdf-48e9-XXCXXXXX

Now, in order to keep the data on elasticsearch against each team in sync with the database, we run a cron once every 24 hours. This cron basically does the following:

  1. Runs for each team of the application in a loop
  2. For each team, it first deletes the existing index from elasticsearch
  3. Then create a brand new index for that team
  4. Then updates the index, for each nested records or documents

While the above process is running, sometimes it does not complete the whole process for a team, and gets stuck while either creating the new index or updating the data of that index.

While monitoring the logs received, we could find following logs that seemed to have some relation with above issue:

Log 1:

(node:9960) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 abort listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit (Use node --trace-warnings ... to show where the warning was created)

Log 2:

Root causes: validation_exception:
Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open;

Log 3:

RequestAbortedError: Request aborted at SniffingTransport.request (/home/netzwelt/Documents/projects/internal/project1/server/node_modules/@elastic/transport/lib/Transport.js:502:46)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at runNextTicks (node:internal/process/task_queues:64:3) at listOnTimeout (node:internal/timers:538:9)
at process.processTimers (node:internal/timers:512:7)
at async Client.UpdateApi [as update] (/home/netzwelt/Documents/projects/internal/project1/server/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
at async setESIndex (/home/netzwelt/Documents/projects/internal/project1/server/src/services/elastic-search/elastic-search.js:66:21)
at async /home/netzwelt/Documents/projects/internal/project1/server/src/helpers/elastic-search.js:339:26
meta: {
body: undefined,
statusCode: 0,
headers: {},
meta: {
context: null,
request: [Object],
name: 'elasticsearch-js',
connection: [Object],
attempts: 0,
aborted: true
warnings: [Getter]

The process mentioned above fails sometimes, and mostly for specific teams, having very large data associated with them.

Few things to note here are:

  1. Other teams having similar size data never have any problem completing the above process successfully.
  2. Elasticsearch logs that we get right now are not descriptive enough to figure out the actual problem.
  3. But once the request fails for one team, other teams are also not indexed properly.

Any guidance or direction to find the root cause of this issue will be highly appreciable.

What is the output of:

GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

If some outputs are too big, please share them on and link them here.

Thank you for the response. Sharing the required information in the gist link below:

As you have only one node, I'd suggest to change the number of replicas to 0 for all the indices.

But as you have a lot of very tiny indices, may I suggest you look at the following resource about sizing:

You have too many small indices IMO.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.