Hi, we have a single node elk cluster running on ubuntu over docker. ELK version is 6.4.3.
We started to get "cluster_block_exception with FORBIDDEN/8/index write api" error on the logstash output. I checked all the issues that possibly cause that error written on the forum posts and articles all around the internet.
- There is enough disk space (around 600 GB)
- There is enough memory and JVM Heap (around 10GB free)
- There is no readonly or write block set neither on index nor on the cluster settings (checked several times and set them false over and over again)
Before the error first came in, I reindexed the largest index on the elk (around 450 GB) to a new index with more shards to gain some indexing speed. (I cannot use split api because of the index settings that blocked me).
After the reindex, I took a snapshot of the old index to another disk. Then I deleted the old index. Because elk started to warn me for 85% of the disk space was used. It appeared to be removed immediately although it has 450 GB of data. I checked the free space from the os. And the warning has gone.
Then I started to reindex again, The reindex speed raised about 10 times comparing the old index. After the indexing 50 - 60 GB of more data, I started to index data to another index on the same elk. And the first error came in to logstash output log. (elastic log are always clean. I have never seen any error message on the elastic output. All the errors appeared in the logstash output)
We've never get that kind of error since we started to use elk for about 1 year. We thought that it is related with the second indexing. Stopped that indexing and restarted the main indexing again. After a while, error came in again.
We tried whole the solutions mentioned all over the internet. But they've never solved the problem indefinitely. Error has gone for a while and then popped up again.
Additionally, I tried to slow down the logstash pipeline (20 event, 100msec delay and single worker) and merge indexes. But they also didn't solve our problem.
Do you have any further idea?