# Recreating an index from LogStash and swapping it via alias **URL:** \ **Category:** Logstash\ **Created:** [May 22, 2017, 10:27am UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652 "2017-05-22T10:27:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Xavi\_Ametller](https://avatars.discourse-cdn.com/v4/letter/x/b5ac83/32.png) [@Xavi\_Ametller](https://discuss.elastic.co/u/Xavi_Ametller)\ **Post date:** [May 22, 2017, 10:27am UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/1 "2017-05-22T10:27:54Z")
Hi there, I would like to periodically generate an index whose data will come from the outcome of a SQL query. Even though I'm aware it is not the traditional use case as the data are not logs nor accumulative, LogStash gives me a good starting point: it makes it very easy to run the query via JDBC and map the data to the index. Ideally the flow would be something like: 1. _Index-A-Yesterday_ already exists and ES alias for _Index-A_ points to that index 2. Call LogStash to create _Index-A-Today_ 3. Warm-up _Index-A-Today_ (optional step) 4. Call ES API to switch _Index-A_ alias to point towards _Index-A-Today_ 5. Delete _Index-A-Yesterday_ Is there a way via LogStash to this? I can't find it 😑... I've seen this topic already asked [here](https://discuss.elastic.co/t/best-practice-for-building-new-version-of-an-index-and-replacing-existing-using-logstash/69819) without a satisfactory answer and since it is closed I try again, worst case I will get an official "not possible". Thanks, Xavi ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 22, 2017, 12:23pm UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/2 "2017-05-22T12:23:50Z")
Not possible in Logstash, currently or perhaps ever. While it is possible to create an index based on a field value, the place where you are likely to hit a wall is aliasing. Logstash has no mechanism at all to rotate aliases. The common misunderstanding is that Logstash actually _creates_ indices at all. It doesn't. This is an important distinction. As you understand the actual flow, then the reason why the flow you described cannot work will become clear. Logstash sends a bulk request to Elasticsearch requesting that a given document (or log line, or DB row, or whatever the event is) be indexed in the index named "logstash -YYYY.MM.dd" (or whatever you have specified for the `index =>`. Elasticsearch interprets these requests, and if the named in the bulk request exists, it puts the document there. If it doesn't exist, Elasticsearch creates the index, and puts the document there. At no point ever does Logstash actually call the `_create` index API. This is the reason why Logstash cannot do alias rotation. It never performs any timed behaviors at all. It may be of benefit for you to reconsider your data storage model to allow for other indexing and management possibilities. One such is the `_rollover` API, which can rollover indices _and_ simultaneously do the alias switching that you've described. The downside of this approach (by itself) is that there's a fractional chance of getting some of the next day's data in "today's" index, or vice versa. But it might be just as useful to not keep daily indices, and let your application do the date filtering for you, rather than rely on date-range indices. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 22, 2017, 12:49pm UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/3 "2017-05-22T12:49:38Z")
Now, with that explanation out of the way, there are potential work-arounds that might make what you want to do possible. One such is to use [Elasticsearch Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/index.html) 1. **Disable [automatic index creation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation)** This approach would simply make Logstash spool up traffic to the new index until some _other_ process created it. This would be safest with Logstash 5.4, which has the official release of [persistent queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html), so the data would safely spool to the local disk on Logstash until "connectivity" was restored to the new index. 2. **Use Curator to [create the new index](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/create_index.html)** This would have to be done via cron to run at exactly 00:00 UTC time, as that's when index rollover happens. 3. **Use Curator to [change the alias](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/alias.html)** This could easily be done as a subsequent step/action in the same configuration file. The downside to this approach is that if something doesn't run right (misconfiguration, for example), you could be spooling data to Logstash's persistent queues for a while. No data would be lost, but it would take a while to empty those queues out if they build up to a large size. ---
**Author:** ![Xavi\_Ametller](https://avatars.discourse-cdn.com/v4/letter/x/b5ac83/32.png) [@Xavi\_Ametller](https://discuss.elastic.co/u/Xavi_Ametller)\ **Post date:** [May 23, 2017, 7:10am UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/4 "2017-05-23T07:10:36Z")
Thanks @theuntergeek for such detailed and clear information! I think I will go for a _custom application/script_ as in our context: - It is not acceptable to mix data from different batches. This discards the `_rollover` approach - I don't know how often will we refresh the data (plus I guess we will change the periodicity in the future). - We want to be able to manually force an update of the index easily (just in case). Last two topics discard the [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/index.html) approach (as it's bound to a specific time). ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 23, 2017, 6:00pm UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/5 "2017-05-23T18:00:49Z")
Well, if you code or script in Python, feel free to use the [Curator API](http://curator.readthedocs.io/en/latest/). This would allow you to re-use the parts that make sense, and code around what doesn't—without having to reinvent the wheel, as it were. ---
**Author:** ![Xavi\_Ametller](https://avatars.discourse-cdn.com/v4/letter/x/b5ac83/32.png) [@Xavi\_Ametller](https://discuss.elastic.co/u/Xavi_Ametller)\ **Post date:** [May 24, 2017, 10:57am UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/6 "2017-05-24T10:57:34Z")
@theuntergeek , appreciate the input but Python is not in our company stack (at least as of now). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 21, 2017, 10:57am UTC](https://discuss.elastic.co/t/recreating-an-index-from-logstash-and-swapping-it-via-alias/86652/7 "2017-06-21T10:57:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.