How to reindex data in Elasticsearch without actually changing index names?

[amruth]

Hi,

I am trying to reindex data in Elasticsearch without actually changing index names. I am ingesting data to indices based on monthly index names(index-%{+YYYY.MM}). I've seen concept of using aliases, but I am not sure how to create new index. Do I need to create index_new or index_new-%{+YYYY.MM}?

Can someone please help me solve the issue?

Thanks

[Bernt_Rostad]

You must reindex into a new index so the name must be different from the old index. An alias is a good way to hide the actual index names from the query clients. That way you can create new versions of an index, call the Reindex API to build up the new index and finally point the alias to the new index.

In my company we also do monthly indices and from time to time we need to reindex because of mapping changes. In order to do that we have a simple rule for mapping an alias to an index:

<alias prefix>_<name>_<yyyy>_<mm>: <index prefix>_<name>_<yyyy>_<mm>_<version>

The alias prefix is a common text string prepended to all our searchable aliases, it was needed to allow clients to search across all aliases (GET /<alias prefix>*/_search?q=...) while skipping over the indices (to avoid duplicate results). The index prefix is also a text string but only prepended to our indices and so only known to our internal systems, typically our indexer engine. This allows us to handle multiple versions of indices and easily change aliases without anyone noticing, e.g.:

POST /_aliases
{
     "actions" : [
           { "remove" : { "index" : "int_blog_2017_04-v3", "alias" : "pub_blog_2017_04" } },
           { "add" : { "index" : "int_blog_2017_04-v4", "alias" : "pub_blog_2017_04" } }
     ]
}'

[amruth]

Hi Bernt. Thanks for your reply.

So right now I have indices with the names "project-2017.08", "project-2017.09", "project-2017.10". In the Kibana end the index looks like "project-*".

• Do I need to create 3 new indices with the names "sample-project-2017.08_v1" , "sample-project-2017.09_v1" , "sample-project-2017.10_v1" and point alias to the new indices?

• If so, what about the next month's data? I am assuming I need to declare index template with required mapping so that next month's data will be distributed to project-2017.11 without any aliases.

I am totally confused how things work for monthly based indices. Please let me know if this is the correct process.

Thanks for the support

[amruth]

Can someone please help me solve the issue?

[dadoonet (David Pilato)]

Please read

About the Elasticsearch category

The heart of the free and open Elastic Stack Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease. PLEASE READ THIS SECTION IF IT'S YOUR FIRST POST Some useful links: elasticsearch reference guide elasticsearch user guide elasticsearch plugins elasticsearch cl…

Specifically the "be patient" part.

[Christian_Dahlqvist]

You should be able to create a new, updated index template that apply to all indices that starts with the pattern project-*. This will apply to all new indices being created, e.g. project-1017.11. If you then reindex the content of project-2017.09 into project-2017.09-1, the template will apply here as well. You should then be able to delete the project-2017.09 index and once this is done create an alias project-2017.09 that points to project-2017.09-1. If you are primarily querying through Kibana, the new index will still match the configured index pattern and you may not even need to create the alias.

[amruth]

I am using Logstash to push data to ES index. Configuration of Logstash would be something like,

output {
                elasticsearch {
                        hosts => ["host1","host2","host3"]
                        index => "project-%{+YYYY.MM}"
                }
}

Can you please say where would this month's data go if I delete project-2017.10(after reindexing data to project-2017.10-1)?

[Christian_Dahlqvist]

If you add the alias, you would still be able to index into this if necessary, and data would end up in the new, linked index.

[amruth]

So for the last 2 months I can just reindex data and delete old indices. But for the present month I will need to add the alias.

   curl -XPOST localhost:9200/_aliases -H 'Content-Type: application/json' -d '
    {
        "actions": [
            { "add": {
                "alias": "project-2017.10",
                "index": "project-2017.10-1"
            }}
        ]
    }

Please let me know if this is the right approach.

[Christian_Dahlqvist]

I have not tested it, but think it should work.

[amruth]

Thanks a lot Christian. I have reindexed everything. It's working now.

This is fine since I have just 3 indices, what if I have 20 indices? Do I need to repeat the process for all 20 indices and delete old indices?

[Christian_Dahlqvist]

Yes, you would need to do it for all indices.

[amruth]

Got it, thanks for your support.