Splitting a single index into daily indices

Elastic Stack Elasticsearch
1

My situation is a little tricky. I currently run Elasticsearch 1.3.2 that has just ONE index over the months.

I need to accomplish two things:

a) Upgrade the version to Elasticsearch 2.3.2
b) Convert that ONE index to daily indices. (Going forward, I want indices to be created on a daily basis.)

How do I proceed? I need some direction as I'm lost.

Many thanks!

2

What programming language are you comfortable with?

What is the size of the current index in terms of documents?

Does each document have sub-document(s)?

Does each document in the current index have a date/time field?

Do you still have the original data files?

What kind of HW do you have? Can you stand up ES 2.3.2 in a separate HW?

3

After upgrading you could use the reindex API. It can take a query param, you could iterate over each day and send them to a different destination using it: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

4

Groovy is what I'm comfortable with.
Size of current index is 10512 documents.
No sub-documents
We do have time stamps
I haven't understood this question about original data files.
We have a Linux box on which we install ELK. I can afford to have ES 2.3.2

5

Since ES is using Lucene under the hood, and each ES version uses a different version of Lucene, I would not try to "move" the existing index into the new ES. You could try backup and restore to see if it works. I would do this as an experiment with an expectation that it won't work.

I don't know anything about Groovy so I can't help you here. If someone on your team can do either python or java, there are libraries that you could use to programmatically get it done quicker (just my personal opinion)

This is not much... you could do it programmatically by reading from one index and write to the new index. I would use the REST API interface, this way your code is neutral to both sides.

Great news. Less thing to worry about.

You can extract the value here to determine which daily index you want to write the data to

Meaning if you have the data files that you indexed into the current index. If you do, you can index these data files into the new ES without worrying about the existing index.

Good. You can write a program to read from one and write to the other. Otherwise, you have to read from existing index, save to a file or a database, shutdown existing ES, bring up new ES, read from files or database and index into the new ES.

The last thing that I suggest you to look into is if the current index, is there any field that was configured to have the data indexed but do not stored. If there is one, you'll probably loose that info when you move the data into the new index

That's all I can think of for now. Others might have different suggestions so wait for a few days to gather for more info before deciding what you would like to do.