Index auto increment issue

Elastic Stack Logstash
1

My requirement is to rollover indices after 7days. I successfully created dynamic index in my logstsh.conf file by using below link
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-aliases.html
Below Is my dynamic index name with my rollover template
index => "coffii_amq-000001"
template => "/etc/logstash_worker/templates/coffii_rollover_logs_template.json"
In my rollover template, I created aliase for my dynamic index with the following script
"aliases" : {
"amq-rollover-alias" : {}
}

then I wrote curator rollover script by the help of below link
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/ex_rollover.html
Below is the curator rollover script
actions:
1:
action: rollover
description: >-
Rollover the index associated with index 'name', which should be in the
form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
disable_action: False
name: amq-rollover-alias
conditions:
max_age: 7d
max_docs: 1000

Everything is working fine but when I run curator script, It successfully created a new index name coffii_amq-000002 but in my logstash.conf file I hard coded name coffii_amq-000001 that’s why after running curator script, System still creating documents in the old index (coffii_amq-000001).Is there a way to define dynamic index in my logstash.conf file. Kindly advise.

2

Configure Logstash to point to the alias:

index => "amq-rollover-alias"

That's what you need to do.

3

If we declare alias as index as you suggested then where we put our index name series (coffii_amq-000001) or should i use both (index and alias) like below this.

index => "coffii_amq-000001"
index => "amq-rollover-alias"

Furthermore should i still retain our index template and in curator script should i use alias(amq-rollover-alias)or index name (coffii_amq-000001).Kindly advise.

4

You defined your index name pattern when you made the alias: amq-rollover-alias and assigned it to index koffii_amq-000001. You need to set your logstash index to the alias. When you do the rollover through Curator, if the conditions you set are matched, Elasticsearch will create index koffii_amq-000002 and then point the alias at that index. This action will be nearly instantaneous. Indexing to the alias will continue without interruption, but the data will flow into the new index.

You do not define a dynamic index in Logstash. You set index => amq-rollover-alias

5

I got it about working in logstash using index => amq-rollover-alias
And also got your suggestion. But my confusion is that how we would define your below suggestion
“You defined your index name pattern when you made the alias: amq-rollover-alias and assigned it to index koffii_amq-000001. ”
My first question is that above suggestion will incorporate into index template. The option I know is only defining alias as shown below
"aliases" : {
"amq-rollover-alias" : {}
}
but don’t know how to tell that this alias is attach in that particular index in the template. Can you please share some code example in template? Or share any link.Thanks.

6

This is a misunderstanding.

You seem to believe that you need to have the rollover alias in the index template. You do not.

Once you have created the incrementable index with the initial alias:

PUT /koffii_amq-000001 
{
  "aliases": {
    "amq-rollover-alias": {}
  }
}

...you don't need the alias in the index template. You should have created the index mapping template first, so that when you created that first index (koffii_amq-000001), the desired mapping was applied at creation time.

Once that incrementable index + alias are created, there is no need to keep re-associating each new index with the alias via the index template. The rollover API keeps pointing the new indices to the existing alias, and unassociating the old index. I repeat: the Rollover API does it all for you after the initial index + alias are created.

Your index template should match the pattern koffii_amq-*, so that all indices created thereafter will get the desired mapping. But you don't need to include any alias data in the mapping template unless you're also mapping to another alias in addition to the rollover alias.

1 Like
7

Thanks a lot Aaron. My issue resolved under your kind guidance.

8

It work for me! The greatest thanks for this post! 3 days could not understand! :slight_smile:

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.