Does elasticsearch automatically close replicated indices?

Elastic Stack Elasticsearch
1

Hey everybody :wave:

I have a new question about the behavior of elasticsearch when it comes to cross cluster replication.
First up, I cannot reproduce the issue, and I don't see any logs regarding an API call for closing an index in the elastic logs.

So, for whatever reason, all our follower indices got closed after we restarted elasticsearch (it seems that it is connected to the restart) - happened on both datacenters (bi-directional setup):

image
image.png821×629 49.7 KB

My question is, is there a situation, where elasticsearch actively closes indices? As I said, I did not find any loglines which state that they got closed by an API call or whatsoever.

Please halp :frowning:

2

Which version are you using?

Can you share all the logs going back before this occurred (i.e. before and after the restart)? I can set up a private upload area if you'd rather not share them in public.

3

Oh yea, we are using 6.8.x.
Yes please, private would be nice!

4

Sure, no probs, you can upload things here: https://upload-staging.elastic.dev/u/619d5b64-bbae-483c-9a86-883e9c767804

5

upload done.
Its the logs of each dc and each node

6

Thanks. These logs contain entries from 2019-10-17T07:57:38,540 until 2019-10-17T19:57:51,579, so a few hours about a week ago Is this when the problematic restart happened?

7

exactly, that was the day all started
The first ClusterBlockException we encountered on client side was around 09:23 UTC that day:

image
image.png1899×377 41.1 KB

8

Are the timestamps in the logs in UTC? The first indication that a node was restarted was logged with a timestamp of 2019-10-17T10:59:05,317.

9

Yes they are utc.
Hmm interesting, I missed that. So it seems that at least one index was closed before the restart already - and there is only one log-entry which states that... I can tell for sure, that the ClusterBlockException occured way more after the restart.
Nevertheless, shouldn't there be a log entry if an index gets closed?

10

Yes, looking at the code I cannot see a path to closing an index that doesn't involve logging a message containing the string closing indices.

Your second screenshot mentions ClusterBlockException[blocked by : [FORBIDDEN/4/index closed];] at 06:19, which precedes the first entry in the log files you've shared (i.e. 07:57). This makes me suspect that these logs do not go back far enough. Do you have earlier logs?

11

Interestingly, the logs from the day before dont say anything at all - except some license warning.

image
image.png1249×614 20 KB

So, No logs before that :frowning:

12

Another possible string that might be logged is recovering as closed after a full cluster restart, but in any case an index becoming closed is something that results in log messages. It's also not something that happens automatically to CCR followers, so we think it must have been an external influence which closed these indices.

13

I'll monitor that cluster more closely in the future - if this happens again without any logs, I'll reply in this forum post again - hopefully with more info.

Thank you so far David!

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.