Cluster currently has [1000]/[1000] maximum normal shards open

Elastic Stack Elasticsearch
1

I have what is like a Newbee question, I hope it makes sense how I explain it.

I noticed that logs are not being ingested by my 1 node cluster and logstash complains about the maximum normal shards open - can you help me identify what I am doing wrong please.

The output I have configured for logstash is directing the logs to this index

index => "vmware_esxi-%{+YYYY.MM.dd}"

and I can see in kibana that the indices are being created and kept open

what I've setup in terms of indexes is this:
index template with Index pattern "vmware_esxi-*"

{
  "index": {
    "lifecycle": {
      "name": "pci_1y"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}

the index for yesterday is about 25.71gb and "GET /vmware_esxi-2024.03.05" is

{
  "vmware_esxi-2024.03.05": {
    "aliases": {},
    "mappings": {
      "properties": {
...
      }
    },
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "provided_name": "vmware_esxi-2024.03.05",
        "creation_date": "1709596800288",
        "number_of_replicas": "1",
        "uuid": "d1J6jTidS2Sp-FsycSKTCw",
        "version": {
          "created": "8060299"
        }
      }
    }
  }
}

the ilm policy is this (obtained via "GET _ilm/policy/pci_1y") - if it says "indices": does that mean this is not applying to any indices?

{
  "pci_1y": {
    "policy": {
      "phases": {
        "warm": {
          "min_age": "30d",
          "actions": {
            "set_priority": {
              "priority": 50
            }
          }
        },
        "cold": {
          "min_age": "90d",
          "actions": {
            "set_priority": {
              "priority": 0
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            },
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "30d"
            }
          }
        },
        "delete": {
          "min_age": "365d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        "vmware_esxi"
      ]
    }
  }
}
2

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

If some outputs are too big, please share them on gist.github.com and link them here.

3

Thanks, this is the output requested:

GET /

{
  "name": "pnl0000vspr4306",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "ejYdH5t2QqWZ3hkF0hIquQ",
  "version": {
    "number": "8.6.2",
    "build_flavor": "default",
    "build_type": "rpm",
    "build_hash": "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date": "2023-02-13T09:35:20.314882762Z",
    "build_snapshot": false,
    "lucene_version": "9.4.2",
    "minimum_wire_compatibility_version": "7.17.0",
    "minimum_index_compatibility_version": "7.0.0"
  },
  "tagline": "You Know, for Search"
}

GET /_cat/nodes?v

ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
10.6.11.104           64          97  70    7.85    6.91     6.17 cdfhilmrstw *      pnl0000vspr4306

GET /_cat/health?v

epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1709764255 22:30:55  elasticsearch yellow          1         1    530 530    0    0      431             0                  -                 55.2%

GET /_cat/indices?v

4

Thanks.

Could you share the logstash logs?

5

If you have a single node cluster you can not have any replicas. You therefore need to set this to 0 in your template and also change the number of replicas to 0 for all existing indices using the update index settings API. Even though Elasticsearch can never allocate these replica shards, they still count towards the maximum number of shards.

The index pattern you have configured indicates that you are not using rollover, but you still have rollover configured in your ILM policy. I think you need to fix this as I do not see how this policy could be working.

6

Things I learned! :sweat_smile:
I was not expecting that but this makes sense.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.