Cross indexes aggregations - should I worry?

Elastic Stack Elasticsearch
1

Hello,

I'm writing an analytics system based on elasticsearch 2.3 and I'm trying to decide which approach to use: one index for all documents OR multi indexes (index per month).
Since I'm doing aggregations on date range (max 6 months back) - I'm afraid It might have performance issues with cross indexes aggregations... I couldn't find an answer on the internet, so I decided to ask here.
I'm expecting to have about 3M documents per month at the beginning.
and I assume that most aggregations will have a date range of 2 months (will have to access 2 indexes in case of multi-index solution).

So, what would you do?

  1. One index of ~90M documents + add deletion plugin for cleanups
  2. Creating 12 indexes (easy monthly documents cleanup)

Thanks in advance

2

I'd never, ever do 1. The cost in doing that delete-by is way too high if you can avoid it.
Go with 2.

The cost of cross querying doesn't matter, take a look at the bottom of this page;

Searching one index that has five primary shards is exactly equivalent to searching five indices that have one primary shard each.

1 Like