I implement logstash elasticsearch output to create daily indices. The index name has this format:
"default-%{[kubernetes][pod][name]}-%{+YYYY.MM.dd}"
It works perfectly, it creates index, lets say for example for pod myapp, default-myapp-2020-10-07. The problem is, I implement elasticsearch ILM, that makes older indices readonly. It is worked, when I check index setings, It has index.blocks.write: ture. But in my logs of elasticsearch, I can see errors like this:
2020-10-07T20:32:27.375602811Z [2020-10-07T20:32:27,375][INFO ][logstash.outputs.elasticsearch][main][1c3200686d9d53e61b23d13f296631b14574b974f7c69a1fa540f2cc7d71ad72] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [default-myapp-2020.10.06] blocked by: [FORBIDDEN/8/index write (api)];"})
2020-10-07T20:32:27.376055814Z [2020-10-07T20:32:27,375][INFO ][logstash.outputs.elasticsearch][main][1c3200686d9d53e61b23d13f296631b14574b974f7c69a1fa540f2cc7d71ad72] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [default-myapp-2020.10.06] blocked by: [FORBIDDEN/8/index write (api)];"})
2020-10-07T20:32:27.376547400Z [2020-10-07T20:32:27,376][INFO ][logstash.outputs.elasticsearch][main][1c3200686d9d53e61b23d13f296631b14574b974f7c69a1fa540f2cc7d71ad72] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [default-myapp-2020.10.06] blocked by: [FORBIDDEN/8/index write (api)];"})
2020-10-07T20:32:27.377749531Z [2020-10-07T20:32:27,376][INFO ][logstash.outputs.elasticsearch][main][1c3200686d9d53e61b23d13f296631b14574b974f7c69a1fa540f2cc7d71ad72] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>3}
Section blocked by: [FORBIDDEN/8/index write (api)];" is because the index is readonly.
What is interesting here, that today date should be 2020-10-07 as you see from logs timestamps, but logstash is trying write to index with date 2020.10.06. I do not understand why. This is happening for more indices.
I am using ELK Stack 7.8.1.