# Time-based index writing and reading **URL:** \ **Category:** Elasticsearch\ **Created:** [December 17, 2012, 7:42pm UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098 "2012-12-17T19:42:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![squirrel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/squirrel/32/2571_2.png) [@squirrel](https://discuss.elastic.co/u/squirrel)\ **Post date:** [December 17, 2012, 7:42pm UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/1 "2012-12-17T19:42:44Z")
I'm a newbie and have spent a day or so reading up on elasticsearch and performance. We are using logstash to parse log lines and sending to graylog2 which writes/reads ES. "time-based indexing" is mentioned a lot on various websites/posts but I have not found any concrete instructions. This post has come closest, but the configuration requires the use of a different index alias for reading and writing, and I don't think graylog2 will do that: [https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J](https://groups.google.com/forum/#!searchin/elasticsearch/time%2420based%2420index%2420alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J) Here's my current guess at doing time-based indexing, for weekly indices: - create a template that will match on all index names, setting up index config, mappings, etc. - once a week run a script that does the following: \*\* create a new index with a name containing the date of next Sunday (to rotate at the start of Sunday) \*\* add the new index to the single alias (for example "graylog") with the following filter: { "range" : { "created\_at" : { "from" : , "to" : \, "include\_upper": false } } } \*\* remove the old index from the alias \*\* delete the old index (archive first?) \*\* optimize the most recently completed index However, I don't think this will work because the filter is not applied on indexing, only searching. (As a general solution there are potential problems which do not occur in the graylog2 case: what if the document does not contain "created\_at"? or the filters do not cover the space, or they overlap? A way around these is to have a default index which would be the most recent.) Does this work? If not what exactly is the way to configure time-based indexing using only a single alias? It's too bad that there isn't more explicit support for tuning ES for logging applications (time series data), as it appears to be a common use case and ES has excellent facilities. -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [December 18, 2012, 7:08pm UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/2 "2012-12-18T19:08:56Z")
You can simply create another alias that will always point to the latest index, and index all records using this alias. Every Sunday, when you add a new index and add it to the list of search aliases, you can flip this index alias to point to the new index. It can be done in the same request. On Monday, December 17, 2012 2:42:44 PM UTC-5, squirrel wrote: > I'm a newbie and have spent a day or so reading up on elasticsearch and > performance. We are using logstash to parse log lines and sending to > graylog2 which writes/reads ES. "time-based indexing" is mentioned a lot on > various websites/posts but I have not found any concrete instructions. > > This post has come closest, but the configuration requires the use of a > different index alias for reading and writing, and I don't think graylog2 > will do that: > > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J) > > Here's my current guess at doing time-based indexing, for weekly indices: > > - create a template that will match on all index names, setting up index > config, mappings, etc. > - once a week run a script that does the following: > \*\* create a new index with a name containing the date of next Sunday (to > rotate at the start of Sunday) > \*\* add the new index to the single alias (for example "graylog") with the > following filter: > { > "range" : { > "created\_at" : { > "from" : , > "to" : \, > "include\_upper": false > } > } > } > \*\* remove the old index from the alias > \*\* delete the old index (archive first?) > \*\* optimize the most recently completed index > > However, I don't think this will work because the filter is not applied on > indexing, only searching. (As a general solution there are potential > problems which do not occur in the graylog2 case: what if the document does > not contain "created\_at"? or the filters do not cover the space, or they > overlap? A way around these is to have a default index which would be the > most recent.) > > Does this work? If not what exactly is the way to configure time-based > indexing using only a single alias? > > It's too bad that there isn't more explicit support for tuning ES for > logging applications (time series data), as it appears to be a common use > case and ES has excellent facilities. -- ---
**Author:** ![squirrel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/squirrel/32/2571_2.png) [@squirrel](https://discuss.elastic.co/u/squirrel)\ **Post date:** [December 18, 2012, 7:44pm UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/3 "2012-12-18T19:44:51Z")
Yes, using 2 aliases that's the way to go, but wondering if there is a _single-alias_ method. Guess I'll look into how to add a "write" elasticsearch\_index\_name config to graylog2. On Tuesday, December 18, 2012 11:08:56 AM UTC-8, Igor Motov wrote: > You can simply create another alias that will always point to the latest > index, and index all records using this alias. Every Sunday, when you add a > new index and add it to the list of search aliases, you can flip this index > alias to point to the new index. It can be done in the same request. > > On Monday, December 17, 2012 2:42:44 PM UTC-5, squirrel wrote: > > > I'm a newbie and have spent a day or so reading up on elasticsearch and > > performance. We are using logstash to parse log lines and sending to > > graylog2 which writes/reads ES. "time-based indexing" is mentioned a lot on > > various websites/posts but I have not found any concrete instructions. > > > > This post has come closest, but the configuration requires the use of a > > different index alias for reading and writing, and I don't think graylog2 > > will do that: > > > > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J) > > > > Here's my current guess at doing time-based indexing, for weekly indices: > > > > - create a template that will match on all index names, setting up index > > config, mappings, etc. > > - once a week run a script that does the following: > > \*\* create a new index with a name containing the date of next Sunday (to > > rotate at the start of Sunday) > > \*\* add the new index to the single alias (for example "graylog") with the > > following filter: > > { > > "range" : { > > "created\_at" : { > > "from" : , > > "to" : \, > > "include\_upper": false > > } > > } > > } > > \*\* remove the old index from the alias > > \*\* delete the old index (archive first?) > > \*\* optimize the most recently completed index > > > > However, I don't think this will work because the filter is not applied > > on indexing, only searching. (As a general solution there are potential > > problems which do not occur in the graylog2 case: what if the document does > > not contain "created\_at"? or the filters do not cover the space, or they > > overlap? A way around these is to have a default index which would be the > > most recent.) > > > > Does this work? If not what exactly is the way to configure time-based > > indexing using only a single alias? > > > > It's too bad that there isn't more explicit support for tuning ES for > > logging applications (time series data), as it appears to be a common use > > case and ES has excellent facilities. -- ---
**Author:** ![squirrel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/squirrel/32/2571_2.png) [@squirrel](https://discuss.elastic.co/u/squirrel)\ **Post date:** [December 18, 2012, 8:44pm UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/4 "2012-12-18T20:44:27Z")
FYI - Looks like the graylog2 folks are already working on a complete solution for time-based indexing support in a future release. On Tuesday, December 18, 2012 11:44:51 AM UTC-8, squirrel wrote: > Yes, using 2 aliases that's the way to go, but wondering if there is a > _single-alias_ method. > > Guess I'll look into how to add a "write" elasticsearch\_index\_name config > to graylog2. > > On Tuesday, December 18, 2012 11:08:56 AM UTC-8, Igor Motov wrote: > > > You can simply create another alias that will always point to the latest > > index, and index all records using this alias. Every Sunday, when you add a > > new index and add it to the list of search aliases, you can flip this index > > alias to point to the new index. It can be done in the same request. > > > > On Monday, December 17, 2012 2:42:44 PM UTC-5, squirrel wrote: > > > > > I'm a newbie and have spent a day or so reading up on elasticsearch and > > > performance. We are using logstash to parse log lines and sending to > > > graylog2 which writes/reads ES. "time-based indexing" is mentioned a lot on > > > various websites/posts but I have not found any concrete instructions. > > > > > > This post has come closest, but the configuration requires the use of a > > > different index alias for reading and writing, and I don't think graylog2 > > > will do that: > > > > > > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J) > > > > > > Here's my current guess at doing time-based indexing, for weekly indices: > > > > > > - create a template that will match on all index names, setting up index > > > config, mappings, etc. > > > - once a week run a script that does the following: > > > \*\* create a new index with a name containing the date of next Sunday (to > > > rotate at the start of Sunday) > > > \*\* add the new index to the single alias (for example "graylog") with > > > the following filter: > > > { > > > "range" : { > > > "created\_at" : { > > > "from" : , > > > "to" : \, > > > "include\_upper": false > > > } > > > } > > > } > > > \*\* remove the old index from the alias > > > \*\* delete the old index (archive first?) > > > \*\* optimize the most recently completed index > > > > > > However, I don't think this will work because the filter is not applied > > > on indexing, only searching. (As a general solution there are potential > > > problems which do not occur in the graylog2 case: what if the document does > > > not contain "created\_at"? or the filters do not cover the space, or they > > > overlap? A way around these is to have a default index which would be the > > > most recent.) > > > > > > Does this work? If not what exactly is the way to configure time-based > > > indexing using only a single alias? > > > > > > It's too bad that there isn't more explicit support for tuning ES for > > > logging applications (time series data), as it appears to be a common use > > > case and ES has excellent facilities. -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 19, 2012, 8:30am UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/5 "2012-12-19T08:30:44Z")
Hello, I think you might find bits from this tutorial useful: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... Although, I think you might be better off with the new store-level compression than compressing source: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... Also, if you want to index in time-based indices, you can do that from logstash directly (no need to go through Graylog). And then you can use Graylog just for searching, I guess. And if you need a single alias for all indices, you can use "\_all". ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Tue, Dec 18, 2012 at 10:44 PM, squirrel [greg.denton@gmail.com](mailto:greg.denton@gmail.com) wrote: > FYI - Looks like the graylog2 folks are already working on a complete > solution for time-based indexing support in a future release. > > On Tuesday, December 18, 2012 11:44:51 AM UTC-8, squirrel wrote: > > > Yes, using 2 aliases that's the way to go, but wondering if there is a > > _single-alias_ method. > > > > Guess I'll look into how to add a "write" elasticsearch\_index\_name config > > to graylog2. > > > > On Tuesday, December 18, 2012 11:08:56 AM UTC-8, Igor Motov wrote: > > > > > You can simply create another alias that will always point to the latest > > > index, and index all records using this alias. Every Sunday, when you add a > > > new index and add it to the list of search aliases, you can flip this index > > > alias to point to the new index. It can be done in the same request. > > > > > > On Monday, December 17, 2012 2:42:44 PM UTC-5, squirrel wrote: > > > > > > > I'm a newbie and have spent a day or so reading up on elasticsearch and > > > > performance. We are using logstash to parse log lines and sending to > > > > graylog2 which writes/reads ES. "time-based indexing" is mentioned a lot on > > > > various websites/posts but I have not found any concrete instructions. > > > > > > > > This post has come closest, but the configuration requires the use of a > > > > different index alias for reading and writing, and I don't think graylog2 > > > > will do that: > > > > [https://groups.google.com/\*\*forum/#!searchin/](https://groups.google.com/**forum/#!searchin/)\*\* > > > > elasticsearch/time$20based$ **20index$20alias/elasticsearch/** > > > > lhoG0gE9Kvo/ez3XXIlsAx0J[https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J](https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J) > > > > > > > > Here's my current guess at doing time-based indexing, for weekly > > > > indices: > > > > > > > > - create a template that will match on all index names, setting up > > > > index config, mappings, etc. > > > > - once a week run a script that does the following: > > > > \*\* create a new index with a name containing the date of next Sunday > > > > (to rotate at the start of Sunday) > > > > \*\* add the new index to the single alias (for example "graylog") with > > > > the following filter: > > > > { > > > > "range" : { > > > > "created\_at" : { > > > > "from" : , > > > > "to" : \, > > > > "include\_upper": false > > > > } > > > > } > > > > } > > > > \*\* remove the old index from the alias > > > > \*\* delete the old index (archive first?) > > > > \*\* optimize the most recently completed index > > > > > > > > However, I don't think this will work because the filter is not applied > > > > on indexing, only searching. (As a general solution there are potential > > > > problems which do not occur in the graylog2 case: what if the document does > > > > not contain "created\_at"? or the filters do not cover the space, or they > > > > overlap? A way around these is to have a default index which would be the > > > > most recent.) > > > > > > > > Does this work? If not what exactly is the way to configure time-based > > > > indexing using only a single alias? > > > > > > > > It's too bad that there isn't more explicit support for tuning ES for > > > > logging applications (time series data), as it appears to be a common use > > > > case and ES has excellent facilities. > > > > > > -- -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 19, 2012, 8:32am UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/6 "2012-12-19T08:32:56Z")
P.S. Forget my comment about the alias - it obviously doesn't do the filtering that you need. On Wed, Dec 19, 2012 at 10:30 AM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)wrote: > Hello, > > I think you might find bits from this tutorial useful: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2012/05/19/elasticsearch-for-logging.html) > > Although, I think you might be better off with the new store-level > compression than compressing source: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > Also, if you want to index in time-based indices, you can do that from > logstash directly (no need to go through Graylog). And then you can use > Graylog just for searching, I guess. > > And if you need a single alias for all indices, you can use "\_all". > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Tue, Dec 18, 2012 at 10:44 PM, squirrel [greg.denton@gmail.com](mailto:greg.denton@gmail.com) wrote: > > > FYI - Looks like the graylog2 folks are already working on a complete > > solution for time-based indexing support in a future release. > > > > On Tuesday, December 18, 2012 11:44:51 AM UTC-8, squirrel wrote: > > > > > Yes, using 2 aliases that's the way to go, but wondering if there is a > > > _single-alias_ method. > > > > > > Guess I'll look into how to add a "write" elasticsearch\_index\_name > > > config to graylog2. > > > > > > On Tuesday, December 18, 2012 11:08:56 AM UTC-8, Igor Motov wrote: > > > > > > > You can simply create another alias that will always point to the > > > > latest index, and index all records using this alias. Every Sunday, when > > > > you add a new index and add it to the list of search aliases, you can flip > > > > this index alias to point to the new index. It can be done in the same > > > > request. > > > > > > > > On Monday, December 17, 2012 2:42:44 PM UTC-5, squirrel wrote: > > > > > > > > > I'm a newbie and have spent a day or so reading up on elasticsearch > > > > > and performance. We are using logstash to parse log lines and sending to > > > > > graylog2 which writes/reads ES. "time-based indexing" is mentioned a lot on > > > > > various websites/posts but I have not found any concrete instructions. > > > > > > > > > > This post has come closest, but the configuration requires the use of > > > > > a different index alias for reading and writing, and I don't think graylog2 > > > > > will do that: > > > > > [https://groups.google.com/\*\*forum/#!searchin/](https://groups.google.com/**forum/#!searchin/)\*\* > > > > > elasticsearch/time$20based$ **20index$20alias/elasticsearch/** > > > > > lhoG0gE9Kvo/ez3XXIlsAx0J[https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J](https://groups.google.com/forum/#!searchin/elasticsearch/time$20based$20index$20alias/elasticsearch/lhoG0gE9Kvo/ez3XXIlsAx0J) > > > > > > > > > > Here's my current guess at doing time-based indexing, for weekly > > > > > indices: > > > > > > > > > > - create a template that will match on all index names, setting up > > > > > index config, mappings, etc. > > > > > - once a week run a script that does the following: > > > > > \*\* create a new index with a name containing the date of next Sunday > > > > > (to rotate at the start of Sunday) > > > > > \*\* add the new index to the single alias (for example "graylog") with > > > > > the following filter: > > > > > { > > > > > "range" : { > > > > > "created\_at" : { > > > > > "from" : , > > > > > "to" : \, > > > > > "include\_upper": false > > > > > } > > > > > } > > > > > } > > > > > \*\* remove the old index from the alias > > > > > \*\* delete the old index (archive first?) > > > > > \*\* optimize the most recently completed index > > > > > > > > > > However, I don't think this will work because the filter is not > > > > > applied on indexing, only searching. (As a general solution there are > > > > > potential problems which do not occur in the graylog2 case: what if the > > > > > document does not contain "created\_at"? or the filters do not cover the > > > > > space, or they overlap? A way around these is to have a default index which > > > > > would be the most recent.) > > > > > > > > > > Does this work? If not what exactly is the way to configure time-based > > > > > indexing using only a single alias? > > > > > > > > > > It's too bad that there isn't more explicit support for tuning ES for > > > > > logging applications (time series data), as it appears to be a common use > > > > > case and ES has excellent facilities. > > > > > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:59am UTC](https://discuss.elastic.co/t/time-based-index-writing-and-reading/10098/7 "2017-07-06T02:59:19Z")