Elasticsearch not updating any data with no error

Elastic Stack Elasticsearch
1

Hello, in Elasticsearch 6.6.2 i'm having this weird problem
where elastic stops performing any modification to data. Any UPDATE/INDEX/DELETE operation
returns OK result but data never change in indexes. I see no errors in logs.
If i restart ES it resumes working correctly, than after a couple of days it's locked again.
All indexes are open and green.

Any hint?
Thanks

2

What exact commands are you sending to check this?
Could you share the outputs as well?

3

Hello @dadoonet, thanks for your reply. The issue is observed in a production application where there are many processes writing to elastic on different indexes continuously from 5s to 1m frequency. We have many tens of instances of application and this issue is observed for first time only on this instance. All processes receive "updated" or "created" results but indexes does not contain written documents when we perform searches. I also see in _cat/indices that documents count is not growing.
For reference, i manually tried deleting a document with:
DELETE localhost:9200/test.index/metrics/test
result:

{
    "_index": "test.index",
    "_type": "metrics",
    "_id": "test",
    "_version": 2,
    "result": "deleted",
    "_shards": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 8480123,
    "_primary_term": 56
}

than (even after minutes):
GET localhost:9200/test.index/_search
still shows me document.
Weird enough, if i try to delete document again i receive 404.

4

I'd recommend upgrading to 6.8 as it might fix the problem. I honestly don't know what is happening.
Unless you have disabled the refresh or change its default value for the index.

5

Can i somewhat check current refresh configuration?

6

Look at the index settings

7

This is what i get. Should i expect to see some additional field if refresh is changed? Currently DB is restarted and operative, i should wait to get it hanged.

"settings": {
            "index": {
                "creation_date": "1628763148510",
                "number_of_shards": "5",
                "number_of_replicas": "1",
                "uuid": "2yfLQd4MSSuSpEt1UNaxSg",
                "version": {
                    "created": "6060299"
                },
                "provided_name": "xxx"
            }
        }
8

Hello, sorry for reopening discussion, I've observed behavior after restart and all documents written during the malfunctioning period were correctly written, in fact, i can see them after restart. It seems that hanging happens only at search level. All searches does not return latest documents (either new or updated) yet they return a freezed version of indexes.
May this help identifying the issue?
Thanks

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.