Max_docs_per_index exceeded

Hello. I use Elasticsearch with Graylog. During this week I noticed that my Elasticsearch server is running out of disk space. It’s configured to store 240 indices. Max_docs_per_index is 20 million documents.

Having checked the size of indices on disk, I quickly noticed that one of them is significantly larger than all others. It contained over 900 mln documents when it finally became inactive. What do you think could be the cause?

It's not the first time it's happening. Last November, one of indices grew to 500 mln docs and then turned inactive and all the next indices didn't exceed the 20 million limit till today.

Graylog 4.2.13 and Elasticsearch 7.10.1 running on Debian.

Thank you.

Elasticsearch 7.10 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

Welcome to our community! :smiley:

That seems to be a graylog setting, not an Elasticsearch one, so I am not 100% we can help here sorry.

Thank you, Mark. You gave me a direction. Here's what I found in Graylog log files:

Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [997]/[1000] maximum shards open;]

I've set cluster.max_shards_per_node to 3000. Is it a valid solution? Is there anything else I should do? It's a single-node ES instance.

It's not ideal as it creates heap pressure.

Should I decrease the number of indices, while raising their size limit, then?

Based on the screenshot it looks like each index has 4 primary shards and generally are around 6GB in size. This is quite small (1.5GB) and will as you see result in a lot of shards. If Graylog limits indices by document count it probably would be reasonable to increase the limit by a factor of between 5 and 10 in order to get the average shard size up.

Thank you.