ElasticSearch Cluster overload , Master removes and adds data node

Elastic Stack Elasticsearch
1

Hi all,

We're facing latency issue with our ES cluster. It started to happen since we rolled out a new service which is sending bulk requests to ES (there are other services which send bulk requests too). On average we have 200 items per bulk request.

Some of bulk requests got rejected because queue was full.

here are the errors we captured in logs:

'class org.elasticsearch.ElasticsearchException] ElasticsearchException[Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$7@2b6261ff on EsThreadPoolExecutor[name = UXZ0Umy/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@252608ca[Running, pool size = 16, active threads = 16, queued tasks = 288, completed tasks = 40348562]]]]'

[2020-05-25T11:09:12,195][INFO ][o.e.c.s.ClusterApplierService] [mrp_f6T] removed {{UXZ0Umy}{UXZ0Umy2QIeGiVpfq5xuOA}{S4UjPVEYQQ--5zX3OtbEKw}{172.31.156.21}{172.31.156.21:9300}{heap=heap25g, zone=eu-central-1b},}, reason: apply cluster state (from master [master {6k0IouY}{6k0IouYQS_ObevrvlhJzug}{5gnp3vXnRaWaq-8lxjQUWg}{172.31.164.24}{172.31.164.24:9300}{heap=heap6g, zone=eu-central-1c} committed version [130793]])

[2020-05-25T11:11:40,684][INFO ][o.e.c.s.ClusterApplierService] [6k0IouY] added {{aKQjMri}{aKQjMribTf-yhf--B38q3A}{k-3-yuVDT06kkmooaxGq9Q}{172.31.147.30}{172.31.147.30:9300}{heap=heap25g, zone=eu-central-1b},}, reason: apply cluster state (from master [master {6k0IouY}{6k0IouYQS_ObevrvlhJzug}{5gnp3vXnRaWaq-8lxjQUWg}{172.31.164.24}{172.31.164.24:9300}{zone=eu-central-1c, heap=heap6g} committed version [130813] source [zen-disco-node-join[{aKQjMri}{aKQjMribTf-yhf--B38q3A}{k-3-yuVDT06kkmooaxGq9Q}{172.31.147.30}{172.31.147.30:9300}{heap=heap25g, zone=eu-central-1b}]]])

We also noticed one data node got removed from cluster by master node.

Did it get removed because of cluster overload?

We have 60 data nodes and 27 client nodes running on ES 6.2.4 version.

I could provide /_nodes/stats in case needed.

could anyone here help me please to identify the root cause?

2

Welcome! :smiley: Sounds like your cluster is overloaded.

What's the output from _cat/nodes and _cat/allocation?

1 Like
3

What is the full output of the cluster stats API?

1 Like
4

Thanks for your reply, @warkolm @Christian_Dahlqvist

_cluster/stats

{
  "_nodes" : {
    "total" : 72,
    "successful" : 72,
    "failed" : 0
  },
  "cluster_name" : ā€œcā€,
  "timestamp" : 1590479983213,
  "status" : "green",
  "indices" : {
    "count" : 24,
    "shards" : {
      "total" : 1080,
      "primaries" : 72,
      "replication" : 14.0,
      "index" : {
        "shards" : {
          "min" : 45,
          "max" : 45,
          "avg" : 45.0
        },
        "primaries" : {
          "min" : 3,
          "max" : 3,
          "avg" : 3.0
        },
        "replication" : {
          "min" : 14.0,
          "max" : 14.0,
          "avg" : 14.0
        }
      }
    },
    "docs" : {
      "count" : 40783631,
      "deleted" : 32425256
    },
    "store" : {
      "size_in_bytes" : 628852477240
    },
    "fielddata" : {
      "memory_size_in_bytes" : 767788992,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size_in_bytes" : 7898019525,
      "total_count" : 25633138190,
      "hit_count" : 10127941773,
      "miss_count" : 15505196417,
      "cache_size" : 253420,
      "cache_count" : 672879520,
      "evictions" : 672626100
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 13559,
      "memory_in_bytes" : 933900545,
      "terms_memory_in_bytes" : 532945347,
      "stored_fields_memory_in_bytes" : 310731072,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 0,
      "points_memory_in_bytes" : 38820618,
      "doc_values_memory_in_bytes" : 51403508,
      "index_writer_memory_in_bytes" : 1608225708,
      "version_map_memory_in_bytes" : 328444,
      "fixed_bit_set_memory_in_bytes" : 138043544,
      "max_unsafe_auto_id_timestamp" : -1,
      "file_sizes" : { }
    }
  },
  "nodes" : {
    "count" : {
      "total" : 72,
      "data" : 45,
      "coordinating_only" : 24,
      "master" : 3,
      "ingest" : 0
    },
    "versions" : [
      "6.2.4"
    ],
    "os" : {
      "available_processors" : 924,
      "allocated_processors" : 924,
      "names" : [
        {
          "name" : "Linux",
          "count" : 72
        }
      ],
      "mem" : {
        "total_in_bytes" : 1887767724032,
        "free_in_bytes" : 36502016000,
        "used_in_bytes" : 1851265708032,
        "free_percent" : 2,
        "used_percent" : 98
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 1074
      },
      "open_file_descriptors" : {
        "min" : 1790,
        "max" : 2377,
        "avg" : 2159
      }
    },
    "jvm" : {
      "max_uptime_in_millis" : 39374668976,
      "versions" : [
        {
          "version" : "1.8.0_191",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "25.191-b12",
          "vm_vendor" : "Oracle Corporation",
          "count" : 72
        }
      ],
      "mem" : {
        "heap_used_in_bytes" : 607855460112,
        "heap_max_in_bytes" : 1529644449792
      },
      "threads" : 11229
    },
    "fs" : {
      "total_in_bytes" : 14898892185600,
      "free_in_bytes" : 13958634631168,
      "available_in_bytes" : 13200116637696
    },
    "plugins" : [
      {
        "name" : "analysis-phonetic",
        "version" : "6.2.4",
        "has_native_controller" : false,
        "requires_keystore" : false
      },
      {
        "name" : "discovery-ec2",
        "version" : "6.2.4",
        "has_native_controller" : false,
        "requires_keystore" : false
      },
      {
        "name" : "repository-s3",
        "version" : "6.2.4",
        "requires_keystore" : false
      },
      {
        "name" : "mapper-size",
        "version" : "6.2.4",
        "has_native_controller" : false,
        "requires_keystore" : false
      },
      {
        "name" : "mapper-murmur3",
        "version" : "6.2.4",
        "has_native_controller" : false,
        "requires_keystore" : false
      }
    ],
    "network_types" : {
      "transport_types" : {
        "netty4" : 72
      },
      "http_types" : {
        "netty4" : 72
      }
    }
  }
}

_cat/nodes

198.65.148.192 46 99 18 3.58 2.98 3.03 d - nYj9MGs
198.65.147.246  6 99 21 3.15 3.05 3.15 d - q6Z7Sj6
198.65.136.30  68 99 18 4.25 3.74 3.57 d - GqqGFMD
198.65.144.73  43 99 18 2.71 3.24 3.49 d - 6v9VNYV
198.65.165.18  20 99 20 3.53 3.45 3.42 d - lUbaa6J
198.65.136.151 18 99 18 3.15 2.94 3.02 d - PSIWLOV
198.65.155.208 32 93  6 0.90 0.58 0.51 - - gasOBRO
198.65.139.212 52 99 18 3.48 3.20 3.07 d - LqmOSz3
198.65.135.37  43 99 19 2.70 2.82 3.31 d - ZeBsgUq
198.65.144.142 20 93  6 1.33 0.70 0.56 - - R7ECqq3
198.65.159.34  46 99 20 3.20 3.45 3.41 d - mrp_f6T
198.65.148.62  14 93  5 0.38 0.40 0.38 - - icFLfiv
198.65.146.72  14 99 17 2.72 3.10 3.25 d - TVxHXes
198.65.143.7   57 94  6 1.07 0.72 0.62 - - U4OPw09
198.65.134.200 61 99 19 2.91 2.72 2.76 d - iUXldNH
198.65.133.177 18 99 19 3.80 3.13 3.07 d - 5yC1api
198.65.135.31  32 99 19 2.64 2.77 2.98 d - zU8jT7I
198.65.138.46  60 99 17 3.04 3.16 3.02 d - uk4rx0C
198.65.154.181 17 99 19 4.79 3.49 3.16 d - Rsqwl3q
198.65.168.111 20 99 17 1.89 2.27 2.52 d - 758-sx4
198.65.161.238 45 94  6 0.36 0.53 0.60 - - a7RXCsv
198.65.148.128 69 94  6 0.98 0.66 0.58 - - Wa4X0l4
198.65.174.60  19 99 17 2.68 2.84 3.01 d - t6fino_
198.65.131.142  7 98  0 0.05 0.01 0.00 m - taH2oV3
198.65.164.24  30 96  0 0.01 0.01 0.00 m * 6k0IouY
198.65.171.76  41 95  6 0.23 0.49 0.53 - - OoqYhGR
198.65.137.61  32 99 18 3.85 3.69 3.42 d - ptf5JB_
198.65.150.28  23 93  6 0.60 0.52 0.52 - - a_2nfSL
198.65.147.30  71 99 19 2.69 2.46 2.64 d - aKQjMri
198.65.131.94  39 94  6 0.31 0.42 0.50 - - BZ16d0x
198.65.132.100 14 99 18 4.17 3.22 2.98 d - uVm8EiX
198.65.144.31  23 93  6 0.40 0.46 0.46 - - 7kuI8uO
198.65.145.44   7 98  0 0.00 0.00 0.00 m - x4MUi1O
198.65.138.186 51 94  7 0.42 0.50 0.54 - - 7gYPeK8
198.65.130.225 46 95  6 0.57 0.51 0.58 - - Te2tCH0
198.65.134.228 61 99 19 2.43 2.86 3.09 d - 6dLVHZD
198.65.130.141 48 99 17 2.55 2.79 2.88 d - PlZ5b3k
198.65.167.227 72 99 17 2.63 2.84 2.91 d - UHuIxb4
198.65.168.42   9 99 18 2.94 2.69 2.87 d - AptE0N3
198.65.146.219 18 99 20 3.71 3.18 3.36 d - GJwmpRk
198.65.150.135 47 94  6 0.43 0.40 0.43 - - z56kTSH
198.65.162.32  66 94  6 0.43 0.58 0.58 - - q3m-WWD
198.65.142.10  31 94  7 0.52 0.54 0.55 - - 0k5XIu3
198.65.167.85  19 99 18 2.70 2.76 2.75 d - XAIJ_ag
198.65.157.183 52 99 18 2.31 2.72 2.88 d - 4_QnWgX
198.65.166.198 57 99 19 2.55 2.77 3.07 d - gX2xbKt
198.65.163.172 26 99 17 2.95 3.19 3.18 d - xAZfPnI
198.65.163.60  56 94  7 0.39 0.59 0.62 - - p1adp2E
198.65.146.10  28 99 19 3.11 3.11 3.13 d - 8ZevyBB
198.65.137.234 18 94  6 0.42 0.51 0.56 - - vJq13mE
198.65.157.252 52 99 19 2.96 3.16 3.12 d - tLvPdwN
198.65.175.122 70 99 21 3.76 3.06 2.94 d - awiST2q
198.65.151.47  50 99 18 3.06 3.44 3.50 d - qNoJmKK
198.65.153.73  23 99 16 1.97 2.22 2.57 d - YWk8Cp1
198.65.144.72  71 99 16 2.81 2.69 2.69 d - SwIP-74
198.65.134.48  19 94  7 0.35 0.50 0.59 - - n0tiZ6d
198.65.138.27  33 99 20 3.46 3.50 3.26 d - YQcN6mB
198.65.165.84  18 99 18 3.16 3.41 3.45 d - W0ofmDI
198.65.170.250 47 99 20 3.32 3.44 3.77 d - VHvJZui
198.65.134.23  63 99 18 3.22 2.85 2.91 d - DK2ZtlD
198.65.162.4   15 94  7 0.23 0.48 0.53 - - 1AJpc_T
198.65.170.103 48 94  6 0.28 0.40 0.50 - - 9O2iHwl
198.65.172.166 16 99 17 3.00 2.42 2.80 d - t9_SmZS
198.65.138.71  10 94  7 0.57 0.48 0.52 - - 3flYsW5
198.65.146.150 52 99 22 4.22 3.75 3.73 d - -3Oiiv_
198.65.136.239 61 99 20 3.33 3.01 3.03 d - rP2MOvH
198.65.167.186 70 94  7 0.67 0.48 0.55 - - 9aQGSZ7
198.65.170.0   39 99 17 2.89 3.31 3.26 d - GjpHSv6
198.65.147.148 63 93  6 0.31 0.44 0.50 - - v0vpv80
198.65.170.174 57 99 19 2.64 3.03 3.00 d - ovw6g8c
198.65.162.85  45 99 19 2.96 3.36 3.31 d - EDfmQBT
198.65.175.237 64 94  6 0.58 0.52 0.51 - - UXXO-RM

_cat/allocation:

24 13.2gb 34.6gb 260.5gb 295.1gb 11 198.65.144.73  198.65.144.73  6v9VNYV
24 13.7gb 35.2gb 259.9gb 295.1gb 11 198.65.165.18  198.65.165.18  lUbaa6J
24 12.6gb 33.9gb 261.1gb 295.1gb 11 198.65.174.60  198.65.174.60  t6fino_
24 12.5gb 33.8gb 261.3gb 295.1gb 11 198.65.175.122 198.65.175.122 awiST2q
24 13.1gb 34.4gb 260.6gb 295.1gb 11 198.65.135.31  198.65.135.31  zU8jT7I
24 12.2gb 33.6gb 261.4gb 295.1gb 11 198.65.162.85  198.65.162.85  EDfmQBT
24 13.4gb 34.7gb 260.3gb 295.1gb 11 198.65.146.10  198.65.146.10  8ZevyBB
24   13gb 34.4gb 260.7gb 295.1gb 11 198.65.137.61  198.65.137.61  ptf5JB_
24 13.7gb 35.1gb 259.9gb 295.1gb 11 198.65.132.100 198.65.132.100 uVm8EiX
24 12.8gb 34.1gb 260.9gb 295.1gb 11 198.65.139.212 198.65.139.212 LqmOSz3
24 13.1gb 34.5gb 260.5gb 295.1gb 11 198.65.134.228 198.65.134.228 6dLVHZD
24   13gb 34.6gb 260.5gb 295.1gb 11 198.65.153.73  198.65.153.73  YWk8Cp1
24 12.4gb 33.8gb 261.3gb 295.1gb 11 198.65.144.72  198.65.144.72  SwIP-74
24 12.4gb 33.9gb 261.2gb 295.1gb 11 198.65.157.183 198.65.157.183 4_QnWgX
24 13.9gb 35.2gb 259.8gb 295.1gb 11 198.65.167.85  198.65.167.85  XAIJ_ag
24   13gb 34.4gb 260.6gb 295.1gb 11 198.65.133.177 198.65.133.177 5yC1api
24 13.1gb 34.5gb 260.6gb 295.1gb 11 198.65.148.192 198.65.148.192 nYj9MGs
24   13gb 34.5gb 260.6gb 295.1gb 11 198.65.163.172 198.65.163.172 xAZfPnI
24 12.8gb 34.1gb   261gb 295.1gb 11 198.65.159.34  198.65.159.34  mrp_f6T
24 13.2gb 34.5gb 260.6gb 295.1gb 11 198.65.151.47  198.65.151.47  qNoJmKK
24 12.8gb 34.2gb 260.9gb 295.1gb 11 198.65.168.111 198.65.168.111 758-sx4
24 12.5gb 33.9gb 261.2gb 295.1gb 11 198.65.146.72  198.65.146.72  TVxHXes
24 13.5gb 34.9gb 260.2gb 295.1gb 11 198.65.138.27  198.65.138.27  YQcN6mB
24 13.2gb 34.7gb 260.4gb 295.1gb 11 198.65.146.150 198.65.146.150 -3Oiiv_
24 12.7gb 34.1gb   261gb 295.1gb 11 198.65.147.246 198.65.147.246 q6Z7Sj6
24 12.8gb 34.2gb 260.8gb 295.1gb 11 198.65.130.141 198.65.130.141 PlZ5b3k
24 12.7gb 34.2gb 260.9gb 295.1gb 11 198.65.147.30  198.65.147.30  aKQjMri
24 12.8gb 34.4gb 260.7gb 295.1gb 11 198.65.167.227 198.65.167.227 UHuIxb4
24 12.8gb 34.1gb   261gb 295.1gb 11 198.65.170.0   198.65.170.0   GjpHSv6
24   13gb 34.3gb 260.7gb 295.1gb 11 198.65.135.37  198.65.135.37  ZeBsgUq
24 13.1gb 34.5gb 260.6gb 295.1gb 11 198.65.134.23  198.65.134.23  DK2ZtlD
24 13.2gb 34.5gb 260.5gb 295.1gb 11 198.65.138.46  198.65.138.46  uk4rx0C
24   13gb 34.4gb 260.7gb 295.1gb 11 198.65.136.151 198.65.136.151 PSIWLOV
24 12.6gb 34.1gb   261gb 295.1gb 11 198.65.165.84  198.65.165.84  W0ofmDI
24 13.1gb 34.4gb 260.6gb 295.1gb 11 198.65.168.42  198.65.168.42  AptE0N3
24 13.3gb 34.6gb 260.5gb 295.1gb 11 198.65.172.166 198.65.172.166 t9_SmZS
24 12.4gb 33.8gb 261.3gb 295.1gb 11 198.65.146.219 198.65.146.219 GJwmpRk
24 12.9gb 34.3gb 260.8gb 295.1gb 11 198.65.134.200 198.65.134.200 iUXldNH
24   13gb 34.5gb 260.6gb 295.1gb 11 198.65.170.174 198.65.170.174 ovw6g8c
24 12.6gb   34gb   261gb 295.1gb 11 198.65.136.30  198.65.136.30  GqqGFMD
24   13gb 34.5gb 260.6gb 295.1gb 11 198.65.154.181 198.65.154.181 Rsqwl3q
24 13.8gb 35.3gb 259.8gb 295.1gb 11 198.65.157.252 198.65.157.252 tLvPdwN
24 12.8gb 34.2gb 260.9gb 295.1gb 11 198.65.166.198 198.65.166.198 gX2xbKt
24 13.3gb 34.7gb 260.4gb 295.1gb 11 198.65.170.250 198.65.170.250 VHvJZui
24 12.7gb 34.1gb   261gb 295.1gb 11 198.65.136.239 198.65.136.239 rP2MOvH
5

Hi @Parviz_Makari

Seems you don't have disk-size problem and ram.percent in node stats demonstrate a high memory usage which is expected based on bulk operations you are sending. Assuming your bulk containing index requests not search, some requests are rejected because of a full queue means your ES cannot support the current indexing rate. It could have different roots based on your index mapping, cluster resources and etc. For example you could have a multiple analysers bound to the index definition which increase the cpu usage or most likely IO latency. I kindly recommend you to have a look at this page here which is trying to help for tuning the indexing speed both on the client (with throttling) and ES side.

1 Like
6

@hym Thank you for your assistance!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.