Elasticsearch 8. Refresh API

Elastic Stack Elasticsearch
1

Hi,
I'm facing an probable issue in ES 8.x.x where the refresh operation has become significantly slower compared to versions 7.x.x ≤ 7.17.
In our setup, each index is created with "refresh_interval": "30s". In version 7.x.x, even with more than 1000 indices, there was no noticeable delay (according to QA).

We recently conducted a test with ES 8:

  • Created an index in ES 8
  • Checked for the index — it wasn't visible (as expected due to the refresh_interval: 30s) Time: 2025-05-30T09:29:55.516Z
  • Called /refresh on ES 8
  • Checked it again right after — index was still not visible Time: 2025-05-30T09:29:57.308Z

Cluster configuration (ES 8):

_nodes":{
      "total":2,
      "successful":2,
      "failed":0
   }
"status":"yellow",
   "indices":{
      "count":679,
      "shards":{
         "total":679,
         "primaries":679,
         "replication":0.0,
         "index":{
            "shards":{
               "min":1,
               "max":1,
               "avg":1.0
            },
            "primaries":{
               "min":1,
               "max":1,
               "avg":1.0
            },
            "replication":{
               "min":0.0,
               "max":0.0,
               "avg":0.0
            }
         }
      }

"nodes":{
      "count":{
         "total":2,
         "coordinating_only":0,
         "data":1,
         "data_cold":0,
         "data_content":0,
         "data_frozen":0,
         "data_hot":0,
         "data_warm":0,
         "index":0,
         "ingest":1,
         "master":1,
         "ml":0,
         "remote_cluster_client":0,
         "search":0,
         "transform":1,
         "voting_only":0
      },
      "versions":[
         "8.17.4"
      ]

"os":{
         "available_processors":8,
         "allocated_processors":8,
         "names":[
            {
               "name":"Linux",
               "count":2
            }
         ],
         "pretty_names":[
            {
               "pretty_name":"Ubuntu 20.04.6 LTS",
               "count":2
            }
         ],
         "architectures":[
            {
               "arch":"amd64",
               "count":2
            }
         ],
         "mem":{
            "total_in_bytes":4294967296,
            "adjusted_total_in_bytes":4294967296,
            "free_in_bytes":504348672,
            "used_in_bytes":3790618624,
            "free_percent":12,
            "used_percent":88
         }
      },
      "process":{
         "cpu":{
            "percent":0
         },
         "open_file_descriptors":{
            "min":608,
            "max":2628,
            "avg":1618
         }
      },
      "jvm":{
         "max_uptime_in_millis":237910025,
         "versions":[
            {
               "version":"23",
               "vm_name":"OpenJDK 64-Bit Server VM",
               "vm_version":"23+37-2369",
               "vm_vendor":"Oracle Corporation",
               "bundled_jdk":true,
               "using_bundled_jdk":true,
               "count":2
            }
         ],
         "mem":{
            "heap_used_in_bytes":1009469104,
            "heap_max_in_bytes":2361393152
         },
         "threads":131
      },
      "fs":{
         "total_in_bytes":59814944768,
         "free_in_bytes":59330650112,
         "available_in_bytes":59297095680
      },

It seems like something changed between ES 7 and ES 8 that is affecting refresh behavior.

Maybe the best course of action is either to add more data nodes to handle the load or do nothing - it is just a standart behaviour.

2

Did the index not exist or did it not show any data? How was it created? Which command did you use to check?

A refresh makes indexed documents serachable but does not have anthing to do with indices being created, so I am a bit confused about your statement.

3

With the help of co.elastic.clients.elasticsearch.indices

try {
            CreateIndexRequest.Builder createIndexBuilder = new Builder();
            createIndexBuilder.index(index);
            createIndexBuilder.settings(settings != null ? settings : getDefaultIndexSettings());

            if (mapping != null) {
                Map<String, Map<String, Object>> mappings = Map.of("mappings", mapping);
                String mappingJson = objectMapper.writeValueAsString(mappings);
                createIndexBuilder.withJson(new ByteArrayInputStream(mappingJson.getBytes()));
            }
            createIndexBuilder.timeout(REQUEST_TIMEOUT);
            long startTime = System.currentTimeMillis();

            CreateIndexResponse createIndexResponse = client.indices().create(createIndexBuilder.build());
            long stopTime = System.currentTimeMillis();
            // cache the index status
            boolean indexStatus = createIndexResponse.acknowledged() && createIndexResponse.shardsAcknowledged();

..........

 private IndexSettings getDefaultIndexSettings() {
        IndexSettings.Builder builder = new IndexSettings.Builder();
        builder.refreshInterval(Time.of(timeBuilder -> timeBuilder.time("30s")));
        builder.numberOfShards("1");
        builder.numberOfReplicas("1");
        if (customIndexAnalyzerSetting.isEnabled()) {
            IndexSettingsAnalysis analysis = getCustomAnalyzer();
            builder.analysis(analysis);
        }
        return builder.build();
    }

/our_index/_search?typed_keys=true&ignore_unavailable=true&expand_wildcards=open%2Cclosed&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true

We created an index (refresh-index=30s) and some documents under it. However, even after calling /refresh the documents are not visible either, but after 10-15 sec, there were visible through ES API.