Creating a single huge index or multiple indexes rolled out with a date

Elastic Stack Elasticsearch
1

I am using Elastic Search Java APIs to create the index, write to index and search for data from these indexes. To attain high availability and archiving. My index may acquire a size of 10mb on daily basis. Is it advisable to create a single index or to create index on a daily basis with a date update on weekly basis. If i have the same index with different dates. How difficult it is to search using elastic search API's using these indexes

I am using Elastic search 1.7.3 currently and i may switch to 2.3 going further.

2

You can set the same Alias on your indices to search over all : Index Aliases | Elasticsearch Guide [5.1] | Elastic

Depending on your datasets, you can imagine to create an index per day, week or month.

3

Don't use a monolithic index.
Do use time based, given the size of the data, a weekly one with only a small shard count would be ideal.

Why not 5.1?

4

5.1 doesn't support AWS deployment. Only 2.3 supports AWS, so only planning to upgrade to 2.3 in next 2 weeks. Does 5.1 supports AWS?

5

What do you mean support AWS?
You can run any ES version on an EC2 instance.

6

@sari.reach I think you're phrasing it backwards. You said "5.1 doesn't support AWS deployment," but what you should have said was "AWS ES cannot deploy version 5.1 yet. AWS ES is only at version 2.3 of Elasticsearch."

What @warkolm is talking about is running your own Elasticsearch instance, self-managed, on an EC2 instance you deploy, rather than the AWS ES managed solution. There are reasons this is good, as there are other limitations to the AWS ES service besides just being behind several versions.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.