Elasticsearch using lots of CPU and disk, flipping to read-only during heavy use

[dpitchford]

I am trying to understand behavior I am seeing from Elasticsearch 6.8 on a production site. During a period of heavy use (indexing large numbers of documents), Prometheus metrics are showing me that ES is using large amounts of CPU and a steadily increasing amount of disk space, but memory usage is only slightly elevated over the norm. In the ES logs, I see many messages like this:

[2023-10-20T15:15:01,932][INFO ][o.e.i.IndexingMemoryController] [elasticsearch-master-0] now throttling indexing for shard [[data_explorer][1]]: segment writing can't keep up

This continues until disk usage hits 95%, at which point the index is set to read-only / allow delete mode as described here and the remaining index requests fail. Disk usage then falls to a level slightly below that from before the indexing started. There are no error messages in the process that is invoking ES until the index is set to read-only mode. After this, the index has to be manually reset out of read-only mode.

I have three main questions:

1. Why does disk usage drop back to its original level after flipping to read-only mode?
2. Why does the read-only setting on the index have to be reset manually if the doc linked above says "the index block is automatically released when the disk utilization falls below the high watermark"?
3. What does the throttling seen in the ES logs mean, and what is causing it? Do the ES docs describe this process anywhere? (I have not been successful in finding an official explanation)

[DavidTurner]

@dpitchford

Elasticsearch 6.8 on a production site

This version is really very old, it passed EOL several years ago. I strongly recommend upgrading to a supported version ASAP.

@dpitchford

• Why does disk usage drop back to its original level after flipping to read-only mode?

I would guess either a segment merge or a translog flush, you'd need to check indices stats to confirm.

@dpitchford

Why does the read-only setting on the index have to be reset manually if the doc linked above says "the index block is automatically released when the disk utilization falls below the high watermark"?

Because the docs you link are for 7.17 and you're using a much older version.

@dpitchford

What does the throttling seen in the ES logs mean, and what is causing it?

The cluster is overloaded (likely IO-bound).