Elasticsearch thread pool rejected

Elastic Stack Elasticsearch
1

Hi,
We are using Elasticsearch 5.6 (Next week we will upgrade to 7.6) for our application, we are using elasticsearch mostly for bulk indexing and much fewer for querying the data.
Our cluster contains 8 nodes. 1 Data node and the others are both data and master eligible.
Each node has 8G RAM (4G from it for heap usage) and 2CPU.
We have 2 daily indexes, daily log index with two shards (Around 500k-1mil documents per day) and daily data index with two shards (Around 4-5 mil documents per day).
Lately we get lot of rejected from BULK and SEARCH thread pools we tried to increase our nodes from 6 to 8 but we still see some rejected operations even it did reduce the rejected rate.
Each node doesn't use high amount of CPU or heap but sometimes the load is going high.
We don't know what is the best way to solve those errors or how to identified the cause.. Increase shard size/ stronger machines/ split to dedicated master and data nodes..

How can we fix the rejected?

Thanks

2

How are you sending the data to Elasticsearch?

3

We are using C# Nest API Bulk method

4

above error basically seen when data types or index mapping is wrong as compare to source index data to destination index data

i think you need , check either your data types or index mapping are correct or not?.

Thanks
HadoopHelp

5

The mapping seems correct.
Those are the errors we get from one logged bulk operation (the log contains more success and failed operations this are just 2 samples from the full log)

{
  "Errors": true,
  "IsValid": false,
  "Items": [
    {
      "Matches": null,
      "Operation": "index",
      "Error": null,
      "Id": "AXDMCn9LrIcxTwmun01C",
      "Index": "data_20200311",
      "IsValid": true,
      "PrimaryTerm": 0,
      "Result": "created",
      "SequenceNumber": 0,
      "Shards": {
        "Failed": 0,
        "Failures": [],
        "Successful": 2,
        "Total": 2
      },
      "Status": 201,
      "Type": "data",
      "Version": 1
    },
    {
      "Matches": null,
      "Operation": "index",
      "Error": {
        "Index": null,
        "Shard": 0,
        "Headers": {},
        "RootCause": null,
        "CausedBy": null,
        "Metadata": {
          "BytesLimit": null,
          "BytesWanted": null,
          "Column": null,
          "FailedShards": [],
          "Grouped": null,
          "Index": null,
          "IndexUUID": null,
          "Language": null,
          "LicensedExpiredFeature": null,
          "Line": null,
          "Phase": null,
          "ResourceId": [],
          "ResourceType": null,
          "Script": null,
          "ScriptStack": [],
          "Shard": null
        },
        "Reason": "rejected execution of org.elasticsearch.transport.TransportService$7@4a5feb40 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@723c2a72[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 77533164]]",
        "StackTrace": null,
        "Type": "es_rejected_execution_exception"
      },
      "Id": "AXDMCn9LrIcxTwmun01D",
      "Index": "data_20200311",
      "IsValid": false,
      "PrimaryTerm": 0,
      "Result": null,
      "SequenceNumber": 0,
      "Shards": null,
      "Status": 429,
      "Type": "data",
      "Version": 0
    }
  ]
}

From the log its look like some operations success and some fail for full bulk thread pool

6

No, it has nothing to do with it.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.