[Logstash-ilm-elasticsearch-output] ERROR - ilm settings in Logstash Elasticsearch output filter

Elastic Stack Logstash
1

Hello all,

I am currently on Elastic Cloud (ESS) 7.9.1

Have been playing around with the ilm settings in the Elasticsearch output plugin in Logstash.

The current setting that has been used and has been working is below:

output {
  elasticsearch {
    hosts => ["https://elastic_host_redacted.cloud.es.io:port_redacted"]
    user => "user_redacted"
    password => "password_redacted"
    action => "create"
    index => "index_name_redacted"
    id => "index_id_redacted"
    ilm_rollover_alias => "clean-up-alias"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "clean_up"
  }
}

This has been working well and ok and have been seeing the roll-over indexes. However, with deletion of the lastest roll-over index. It doesn't seem to reappear. Even after a restart of the Logstash agent, refreshing the list of indexes etc.
Through testing also, have played around with added the ilm.enabled = > true in the Logstsash config but this does not seem to help anything. Would this also have an effect now with the settings or cache?

The Logstash index roll ups have been in use for a while and managed to get up to number 10 in the roll-over count. The number 10 index was the one that was deleted.. so currently the latest one is number 9. Like such:

  • clean-up-alias-2020.08.05-000001
  • clean-up-alias-2020.08.09-000002
    ...
  • clean-up-alias-2020.09.18-000009
  • clean-up-alias-2020.09.22-000010

The number 10 index roll-over hasn't re-appeared.

I have tried this with Filebeat and Auditbeat also, with the settings in both of there .yml files as such:

#------------------------
# ILM management
#------------------------
setup.ilm.enabled: true
setup.ilm.rollover_alias: "beats-clean-up"
setup.ilm.pattern: "{now/d}-000001"

these indexes come straight back after a restart.

Am I missing something?
Would it be because the index has been deleted it will not appear again due to the cache or something? So in turn would it mean that the ILM policy will have to be removed and then re-created? Would I need to go in to the number 9 index roll-over, enable it again and then let it roll-over again to recreate? Is this possible?

I would like to recover this back to the normal way it was working but I have a feeling I may have to start again with the ILM settings and indexing?

Any help or point in the correct direction would be greatly appreciated! :slight_smile:
Thank you!

2

When you delete the index, do you also restart Logstash?

3

Hey @warkolm

Thank you for your reply.

Yes, after the index was deleted. Logstash was restarted.
Have gone through on different occasions to restarts, force-stop and restart, force-stop and start, stop and start etc.

4

Ok, then that means it will create a new index pattern of clean-up-alias-2020.09.28-000001, or whatever date you actually restarted Logstash.

5

Hey @warkolm, Thank you for your reply.

This is what I would have thought, and this is what the normal behavior would be like.

However, there is still no re-appearance of the index. There is an index that is running in junction with the index in question and that is all working well and ok. It has performed another roll-over on the other index. But the index in question still isn't showing anything.

When looking on the machine where the Logstash agent is, the agent logs are showing as well with no errors or warnings and shows that the data is/ has been being sent to ELK.

I am a little confused as to why this is happening.

I think I am going to see if I am able to change/ push it to work again today. If not then I suppose I will just bite the bullet and wipe and re-index if needed.

Will keep an update on here with what I find out.

Thank you!

6

As an Update about this issue.

It came to light that there was a change in the elasticsearch output config. As it says in the Logstash elasticsearch output page. When changing these elements, in regards to ilm policy, the index will need to be re indexed. So when having to `remove ALL the previous data, the indices appeared again and now have been showing data in the index pattern.

Link to the ilm policy settings for the Logstash Elasticsearch output plugin -> https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm_enabled

On a note, this was little bit of a strange behavior as when even on a restart, the index should be able re-appear. Will keep an eye on this and see if anything on this will change again or anything is worth noting.

7

Ahh! Another way in which I had forgotten to add to my last update! this is the actions that I performed to gain back the normal/ correct behavior.

Another way this could be done to preserve the data also is to use the reindex API
link to reference on how to use this API is here -> Reindex API

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.