Redirect logs to newly created Index by Index Roll over

Elastic Stack Elasticsearch
1

Hi All
I am using Elasticsearch 7.3 for my internal alerting module. I am planning to introduce the ILM policy in the data node to handle the high amount of data. I am facing an issue I am using custom logstash output, for example, index-01.while the rollover process the index name will change to index-000002.In my case I can't change the logstash output index name, again and again, .how can I redirect the logs to the newly created index with the new name.

Usually, I observed in other tools they will create a new file and move data to that one and the primary one will be static.

Please advise me on this

2

Hello @Ozil

You have 2 options.

Use Logstash native ILM support

output {
      elasticsearch {
        ilm_rollover_alias => "yourcustomindexprefix"
        ilm_pattern => "000001"
        ilm_policy => "custom_policy"
      }
    }

Where:

  • yourcustomindexprefix cannot be a variable (e.g. %{[@metadata][index]})
  • you already defined the custom_policy ILM Policy (via Kibana UI or via API)

Do not use the Logstash native ILM support

output {
      elasticsearch {
        index => "yourrolloveralias"
        ilm_enabled => false
      }
    }

Where:

  • yourrolloveralias is a rollover alias you've bootstrapped after following all those steps, which means:
    • create the ILM policy
    • create an Index Template which makes use of the ILM Policy and specifies the index.lifecycle.rollover_alias name
    • bootstrap the rollover alias
3

@Luca_Belluccini Thank you so much for your support.i think my issue will be resolved by this help.

Thanks for your quick reply sir.

4

Hi @Luca_Belluccini
i am getting an error like "elasticsearch - Unknown setting 'ilm_enabled' for elasticsearch"

my configuration is

output {
   elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "testdata"
        ilm_enabled => false
}
}
5

On which Logstash version are you? It must be 6.7 or more recent

  • Please check the documentation
  • Ensure you're running an Elasticsearch on a Basic license
6

i am using logstash version 6.4.2

7

To setup rollover with ILM, you need at least the Elastic Stack on 6.7.
You can use Logstash 6.8.x or 7.3 to write to Elasticsearch 7.3.

8

Ok i understand . Thank you so much

9

@Luca_Belluccini Hi Sir

I started using the logstash output with the Alias indexing method. but when a new index is created using the rollover process logstash getting stuck. In testing, I can restart the logstash but how I can proceed when I take this to production

Please advise on this

10

What is the configuration of the Logstash output?

Please ensure you're following the steps at

https://www.elastic.co/guide/en/elasticsearch/reference/current/using-policies-rollover.html

11 
output {
stdout { codec => rubydebug }
   elasticsearch {
        hosts => ["http://10.55.50.105:9200"]
        index => "aliasdata"
        ilm_enabled => false
}
}
~~~
12

This is how my Logstash output look like

13

Thank You so much for the replay. I am using Curator yml to perform the rollover action

14

If you're using ilm_enabled => false and you want to use manual Rollover using Curator, you have to ensure the following:

  1. Check there is no index template which matches aliasdata index.

If you're in doubt, please share the output of GET _templates

  1. Create an Index Template, if necessary, matching the index name data-*

  2. Bootstrap the index & write alias using.

In the example below, the index names will be data-000001, data-00002...

PUT /data-000001 
{
  "aliases": {
    "aliasdata": { "is_write_index": true }
  }
}

  1. Start logstash

  2. In Curator, use the rollover action (documentation).

Example (change the conditions to match what you need):

action: rollover
options:
  name: aliasdata
  conditions:
    max_age: 1d
    max_docs: 1000000
    max_size: 5gb

This should be enough to make it work.

15

Thank you @Luca_Belluccini for the detailed information. I am following the process that you mentioned above .will check closely .
thank you for the advises