Hi team,
I am facing an issue while using an Elasticsearch update query. I have given the detailed description of the issue below:
Version of Elasticsearch: 7.17.0
Problem Description:
My application has different teams. For each team, there is some data that is stored on elasticsearch with an index name format as given below:
[data-name]-[team-id]
Eg. attributes-2eXX7a00-XXdf-48e9-XXCXXXXX
Now, in order to keep the data on elasticsearch against each team in sync with the database, we run a cron once every 24 hours. This cron basically does the following:
- Runs for each team of the application in a loop
- For each team, it first deletes the existing index from elasticsearch
- Then create a brand new index for that team
- Then updates the index, for each nested records or documents
While the above process is running, sometimes it does not complete the whole process for a team, and gets stuck while either creating the new index or updating the data of that index.
While monitoring the logs received, we could find following logs that seemed to have some relation with above issue:
Log 1:
(node:9960) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 abort listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit (Use node --trace-warnings ... to show where the warning was created)
Log 2:
Root causes: validation_exception:
Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open;
Log 3:
RequestAbortedError: Request aborted at SniffingTransport.request (/home/netzwelt/Documents/projects/internal/project1/server/node_modules/@elastic/transport/lib/Transport.js:502:46)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at runNextTicks (node:internal/process/task_queues:64:3) at listOnTimeout (node:internal/timers:538:9)
at process.processTimers (node:internal/timers:512:7)
at async Client.UpdateApi [as update] (/home/netzwelt/Documents/projects/internal/project1/server/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
at async setESIndex (/home/netzwelt/Documents/projects/internal/project1/server/src/services/elastic-search/elastic-search.js:66:21)
at async /home/netzwelt/Documents/projects/internal/project1/server/src/helpers/elastic-search.js:339:26
{
meta: {
body: undefined,
statusCode: 0,
headers: {},
meta: {
context: null,
request: [Object],
name: 'elasticsearch-js',
connection: [Object],
attempts: 0,
aborted: true
},
warnings: [Getter]
}
}
The process mentioned above fails sometimes, and mostly for specific teams, having very large data associated with them.
Few things to note here are:
- Other teams having similar size data never have any problem completing the above process successfully.
- Elasticsearch logs that we get right now are not descriptive enough to figure out the actual problem.
- But once the request fails for one team, other teams are also not indexed properly.
Any guidance or direction to find the root cause of this issue will be highly appreciable.