All Elasticsearch Indices in red(25%) & yellow(75%)

[ManimaranS (Manimaran Subramaniyan)]

Hi
I'm using 6.6.2 version with 3 nodes, having 1 as master. The uploading indices are becoming red, whereas the existing ones are in Yellow(75%) and red(25%). Can i have some help to resolve this. Let me know if you need any additional details.

[Christian_Dahlqvist]

What is the specification of your cluster? How many indices and shards do you have in the cluster? Is there anything in the logs that gives a clue about the cause of the problem?

[ManimaranS (Manimaran Subramaniyan)]

Hi
I'm using 6.6.2. Total indices are about 200 and all are replicas with all nodes of the cluster. One of the node among the 3, is with 6.7, will that be an issue for this? Also from the logs, could see some exceptions like "java.io.IOException: Disk quota exceeded" and "[ELK_PROD_1] [[logstash-2019.02.20][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [logstash-2019.02.20][0]: Recovery failed on {ELK_PROD_1}{8ah6TcboTX-NrOxmOy4_IQ}{EAmDDdBxSS6U4Em7B8Vt_A}{10.90.151.39}{10.90.151.39:9300}{ml.machine_memory=16170377216, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}"

[harshbajaj16 (Harsh Bajaj)]

Hi @ManimaranS,

In logs its hsowing Disk Quota Exceëded". Have you checked the same that there is space availbale or not??

java.io.IOException: Disk quota exceeded" and "[ELK_PROD_1] [[logstash-2019.02.20][0]] marking and sending shard failed due to [failed recovery]

Please check and confirm.

However, could i know the reason of using different version in your cluster??

Regards,
Harsh Bajaj

[ManimaranS (Manimaran Subramaniyan)]

Hi @harshbajaj16

Have checked the disk space and could see we have more space with respect to storage. But on the RAM, we are running at 16G per node(each node is deployed to a server). The same is divided as below

1. elasticsearch --> 8G
2. logstash --> 2G
3. filebeat --> 2G

And to answer your question on the different version, it has been installed accidentally and till 3 - 5 months we have no issues on the indices. Can you please confirm, if we can downgrade without any impact to the data?

Note, the size of the log being uploaded is about 15 - 17 G per day

[harshbajaj16 (Harsh Bajaj)]

Hi @ManimaranS

This error usually occurs when the disk space is full but as you confirmed that disk space is available.
please make sure that you have enough disk space dedicated to Elasticsearch, all the time. This does not only mean enough disk space for the data but also for merges going on.

Have checked the disk space and could see we have more space with respect to storage.

Yes, We can downgrade the same but first we need to resolve above error.

if we can downgrade without any impact to the data?

Please clear following point:
Is this error on 6.6.2 machine or 6.7 machine??
could you please share the cluster health status and also share the shards status where it shows red??

Regards,
Harsh Bajaj

[Christian_Dahlqvist]

All nodes need to use the same version and you should ideally have 3 master eligible nodes.

[harshbajaj16 (Harsh Bajaj)]

HI @Christian_Dahlqvist,

My understanding was same but as he mentioned that he was not facing the issue with this setup as he is using the same since last 3-4 months.

it is hard to understand the issue is coming because of the version.