Hi
I'm using 6.6.2 version with 3 nodes, having 1 as master. The uploading indices are becoming red, whereas the existing ones are in Yellow(75%) and red(25%). Can i have some help to resolve this. Let me know if you need any additional details.
What is the specification of your cluster? How many indices and shards do you have in the cluster? Is there anything in the logs that gives a clue about the cause of the problem?
Hi
I'm using 6.6.2. Total indices are about 200 and all are replicas with all nodes of the cluster. One of the node among the 3, is with 6.7, will that be an issue for this? Also from the logs, could see some exceptions like "java.io.IOException: Disk quota exceeded" and "[ELK_PROD_1] [[logstash-2019.02.20][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [logstash-2019.02.20][0]: Recovery failed on {ELK_PROD_1}{8ah6TcboTX-NrOxmOy4_IQ}{EAmDDdBxSS6U4Em7B8Vt_A}{10.90.151.39}{10.90.151.39:9300}{ml.machine_memory=16170377216, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}"
Hi @ManimaranS,
In logs its hsowing Disk Quota Exceëded". Have you checked the same that there is space availbale or not??
Please check and confirm.
However, could i know the reason of using different version in your cluster??
Regards,
Harsh Bajaj
Have checked the disk space and could see we have more space with respect to storage. But on the RAM, we are running at 16G per node(each node is deployed to a server). The same is divided as below
- elasticsearch --> 8G
- logstash --> 2G
- filebeat --> 2G
And to answer your question on the different version, it has been installed accidentally and till 3 - 5 months we have no issues on the indices. Can you please confirm, if we can downgrade without any impact to the data?
Note, the size of the log being uploaded is about 15 - 17 G per day
Hi @ManimaranS
This error usually occurs when the disk space is full but as you confirmed that disk space is available.
please make sure that you have enough disk space dedicated to Elasticsearch, all the time. This does not only mean enough disk space for the data but also for merges going on.
Yes, We can downgrade the same but first we need to resolve above error.
Please clear following point:
Is this error on 6.6.2 machine or 6.7 machine??
could you please share the cluster health status and also share the shards status where it shows red??
Regards,
Harsh Bajaj
All nodes need to use the same version and you should ideally have 3 master eligible nodes.
My understanding was same but as he mentioned that he was not facing the issue with this setup as he is using the same since last 3-4 months.
it is hard to understand the issue is coming because of the version.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.