I am currently working on a project which requires indexing substantial
amounts of timeseries event data. Based on the feedback from other
projects doing similar work (i.e. Logstash), I've decided to try rolling
indices, with a single active index being written at any given time. This
has all sorts of advantages w.r.t. rapid cleanup of old content, and the
ability to optimize historical data for querying, as well as separating
indexing and query hardware into separate zones with appropriate hardware.
I would like to aggregate older indices, in order to prevent excessive
shard allocations. Given that the mappings are identical for all indices,
this should be a fairly efficient operation - a higher level segment merge
across multiple Lucene shards. However, I haven't been able to determine
whether ElasticSearch has any built in support for this currently.
Note that Lucene provides a standalone tool for just this purpose:
So, my question is:
- Does ES already support this behavior in some fashion? Unfortunately
it's very difficult to search for answers to this problem, as 'merge' is a
very overloaded word
- Is there already a roadmap to include similar functionality?
- If there's nothing currently planned, how viable would it be to get this
functionality integrated into ES? I am happy to start working on an
implementation if that is what's needed.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
For more options, visit https://groups.google.com/groups/opt_out.