Indexing stops when cluster is Red?

Elastic Stack Elasticsearch
1

Hi all.

Maybe this is expected behavior, but I would like to change it if possible. I have a 6 node cluster in prod, and whenever the cluster is Red, indexing stops until it changes to Yellow or Green. Is this normal?

I index about 10k events/sec, and today I closed an index (no problems), then opened it again, and it took about 15 minutes to initialize the primary shards, and the whole time that was happening, indexing completely stopped.

I would like the cluster to continue indexing all my other indexes, even if one or two are initializing.

Can someone suggest a configuration setting change that I might try that would allow this to happen?
Thanks so much!
Chris

2

Take a look at https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#no-master-block and see if that helps.

3

Thank you very much, Mark.

I took a look at that link. In my 6 node cluster, all nodes serve all roles, so I have 6 masters. I have discovery.zen.minimum_master_nodes set to 4, so in my case of just one index being Red, I still have both at least one active master, and the minimum master nodes setting is satisfied as well.

I'm not sure this setting would change anything for me in this case, but perhaps I'm missing something?

Here is my complete elasticsearch.yml file:

action:
  disable_delete_all_indices: true
bootstrap:
  mlockall: true
cluster:
  name: elasticsearch-prod
  routing:
    allocation:
      disk:
        watermark:
          high: 98%
          low: 95%
discovery:
  zen:
    minimum_master_nodes: 4
    ping:
      multicast:
        enabled: false
      unicast:
        hosts: 10.200.116.68,10.200.116.69,10.200.116.70,10.200.116.71,10.200.116.72,10.200.116.73
gateway:
  recover_after_nodes: 6
http:
  cors:
    allow-credentials: true
    allow-origin: /.*/
    enabled: true
  enabled: true
index:
  codec:
    bloom:
      load: false
  merge:
    policy:
      max_merge_at_once: 4
      max_merge_at_once_explicit: 4
      max_merged_segment: 1gb
      segments_per_tier: 4
      type: tiered
    scheduler:
      max_thread_count: 1
      type: concurrent
  number_of_replicas: 1
  number_of_shards: 6
  refresh_interval: 60s
  search:
    slowlog:
      threshold:
        fetch:
          info: 1s
          warn: 2s
        index:
          info: 5s
          warn: 10s
        query:
          info: 5s
          warn: 10s
  translog:
    flush_threshold_size: 1GB
indices:
  breaker:
    fielddata:
      limit: 35%
    request:
      limit: 30%
    total:
      limit: 70%
  fielddata:
    cache:
      size: 30%
  memory:
    index_buffer_size: 10%
  store:
    throttle:
      type: none
marvel:
  agent:
    exporter:
      es:
        hosts: 10.200.116.74:9200,10.200.116.75:9200
network:
  host: 10.200.116.71
node:
  data: true
  master: true
  name: elasticsearch-prodes04
path:
  data:
      - /derby/data/disk1/elasticsearch
script:
  disable_dynamic: false
threadpool:
  bulk:
    queue_size: 500
  search:
    queue_size: 2000

Again, thank you so much for your time.
Chris

4

Just bringing this back up again, as I'm still struggling with it. I'm guessing that this is not expected behavior, correct? :slight_smile:

If anyone might have a suggestion as to what could be going on, I would be most appreciative to hear about it.

Many thanks in advance,
Chris