I am planning to use elasticsearch (ES) for storing event logs. Per day,
the application should store nearly 3000+ events and size will be around
30-50K.
I need to take some statistics monthly, half yearly, yearly.... also year
old data can be ignored sometimes but data should be retained for many years
I want to know the best practices in this scenario
Is it a good idea to create shards based on size/period or create one
shard with multiple alices based on filter condition?
Does ES merges the search results coming from multiple shards? If yes,
how fast or better it is compared to lucene's ranking based on Vector space
model?
On Friday, March 21, 2014 10:11:53 AM UTC+5:30, Chetana wrote:
I am planning to use elasticsearch (ES) for storing event logs. Per day,
the application should store nearly 3000+ events and size will be around
30-50K.
I need to take some statistics monthly, half yearly, yearly.... also year
old data can be ignored sometimes but data should be retained for many years
I want to know the best practices in this scenario
Is it a good idea to create shards based on size/period or create one
shard with multiple aliases based on filter condition?
Does ES merges the search results coming from multiple shards? If yes,
how fast or better it is compared to lucene's ranking based on Vector space
model?
Is it a good idea to create shards based on size/period or create one
shard with multiple alices based on filter condition?
I would recommend on using time-based indices, you can hear about the
rationale at Elasticsearch Platform — Find real-time answers at scale | Elastic (the
part you are interested in starts at 21:15 but I would recommend wathing
the whole video which gives interesting ideas about how to model data with
Elasticsearch). For example, you could imagine build monthly indices. Then
tools like curator can help you manage old indices, in order to force-merge
(optimize) the read-only onces and deletes the old ones.
Does ES merges the search results coming from multiple shards? If yes,
how fast or better it is compared to lucene's ranking based on Vector space
model?
Indeed, Elasticsearch merges search results: each shard returns its top
hits (via Lucene) and the node that coordinates search execution merges
these per-shard top-hits based on a priority queue.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.