Docs count delay - docs added regularly, but docs count 0

[jb79]

Hi,

Docs added regularly, but the count (after all added) could be verified after several minutes :
Added docs ( @timestamp from logstash):

Transport traffic

Indexing rate:

Docs count

Seems that sth wrong with indexing - couldn't however find neither error, nor suspicious log in Elasticsearch (all master,client, data nodes checked).

• No parallel docs deletions on that index during insertions.
• All docs are in 1 segment ("num_committed_segments" : 6 - with several docs in other segments from other indices).

      "shards" : {
        "0" : [
          {
            "routing" : {
              "state" : "STARTED",
              "primary" : true,
              "node" : "SwYLLQLhQJqSZJz4uQDcjw"
            },
            "num_committed_segments" : 6,
            "num_search_segments" : 6,
            "segments" : {
...
              },
              "_2" : {
                "generation" : 2,
                "num_docs" : 211097,
                "deleted_docs" : 0,
                "size_in_bytes" : 24433952,
                "memory_in_bytes" : 9796,
                "committed" : true,
                "search" : true,
                "version" : "8.7.0",
                "compound" : true,
                "attributes" : {
                  "Lucene87StoredFieldsFormat.mode" : "BEST_SPEED"
                }
...

What could be the reason of such behaviour ?
What can I check (nodes stat, metric, setting, disk I/O,... )?
On grafana couldn't find anything suspicious - there's also as in the docs count picture above a docs pick visible

Elasticsearch details

  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_date" : "2021-12-20T10:05:59.588364Z",
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },

ES cluster :

{
  "cluster_name" : "abc-elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 19,
  "number_of_data_nodes" : 10,
  "active_primary_shards" : 25,
  "active_shards" : 58,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Thanks in advance!

[jb79]

Another thing we observed was that sometimes docs are loaded first to the replica shard and then to the primary one... :

index shard prirep state docs store ip node
test-index-2022.04.29-000001 0 p STARTED 27242 3.4mb 192.168.23.12 elasticsearch-data-1
test-index-2022.04.29-000001 0 r STARTED 29241 4.1mb 192.168.25.29 elasticsearch-data-2

What's interesting GET test-index/_search returned docs, although before the /_count query returned 0. After /_search query, /_count starts returning > 0 result.