Elasticsearch showing Errors when duing bulk operations?

[Yaswanth]

Hi All,

I am using ES 2.4.1 Cluster hosted in Azure . I have 3 Nodes which are running in 3 VM's each has 4 cores with 14gb RAM and in that i had given 7021m to Elasticsearch HEAP. I am getting EsRejectedExecutionException in my elasticsearch logs . Node 0 logs are here . I can see same errors with different index names in all the other nodes too.

1)Why i am getting these erros and How can i resolve this ?

By search in google i am came to know that this will come if bulk requests increase the bulksize queue . And i also went through the documentation where they have specifed bulk 2) For bulk operations. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors. So for my ES cluster has queue size of (5 * 200 =1000 requests) for each node . But when i checked in my ES Cluster i can see ES is not using all the 4 cores of the VM it is using only 1 . Using azure template i had installed ES in the VM .

{
  "cluster_name": "EScluster",
  "nodes": {
    "iTV34exDQVm8zJHBn5o77g": {
      "name": "data-1",
      "transport_address": "10.0.0.7:9300",
      "host": "10.0.0.7",
      "ip": "10.0.0.7",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.0.0.7:9200",
      "attributes": {
        "max_local_storage_nodes": "1",
        "master": "true"
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Linux",
        "arch": "amd64",
        "version": "3.19.0-65-generic",
        "available_processors": 1,
        "allocated_processors": 1
      }
    },
    "u7_4qXHeTTGdfHVq9lsjiA": {
      "name": "data-2",
      "transport_address": "10.0.0.6:9300",
      "host": "10.0.0.6",
      "ip": "10.0.0.6",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.0.0.6:9200",
      "attributes": {
        "max_local_storage_nodes": "1",
        "master": "true"
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Linux",
        "arch": "amd64",
        "version": "3.19.0-65-generic",
        "available_processors": 1,
        "allocated_processors": 1
      }
    },
    "vFfHEIg6T8OPl3U_L9vLdg": {
      "name": "data-0",
      "transport_address": "10.0.0.8:9300",
      "host": "10.0.0.8",
      "ip": "10.0.0.8",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.0.0.8:9200",
      "attributes": {
        "max_local_storage_nodes": "1",
        "master": "true"
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Linux",
        "arch": "amd64",
        "version": "3.19.0-65-generic",
        "available_processors": 1,
        "allocated_processors": 1
      }
    }
  }
}

3. But in my logs i can see some are having pool size = 4 with queue capacity = 50 others are having pool size = 7 with queue capacity = 1000 . I dont know why i am getting these pictures?

Please let me know if you want any further information.

Thanks

[polyfractal (Zachary Tong)]

The EsRejectedExecutionException exception basically means you are sending requests faster than your node/cluster can process them. It could be due to a very fast burst of requests that overwhelm the node temporarily, or that you are just sending too many requests consistently.

The short answer is that you need to either identify the bottleneck and fix it by adding more resources (disk IO, CPU, etc), add more nodes, optimize your indexing/queries, or just send fewer requests.

Now, as to your queue questions, it's possible your Azure VM is not detecting the processor count correctly. If you know you have 4 cores but only 1 is showing up in the stats, you can use the processors setting to manually configure the core count:

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html#processors

Different VM architectures sometimes mislead or mess up the reported number of cores, which is why we allow the user to override it