# Reindex daily indices to weekly indices

**URL:** <https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780>\
**Category:** Elasticsearch\
**Created:** [August 5, 2019, 11:14am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780 "2019-08-05T11:14:58Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Praneet\_Khandelwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praneet_khandelwal/32/40565_2.png) [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\
**Post date:** [August 5, 2019, 11:14am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/1 "2019-08-05T11:14:58Z")

</div>

I am working on a reindexing problem in which I need to reindex the daily indices to weekly indices. I am using the reindex APi and trying to work with the script processor to make the changes. Can anyone help with what the script might look it?

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [August 6, 2019, 8:57am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/2 "2019-08-06T08:57:26Z")

</div>

I have done something similar lately. While my scripts won't directly solve your problem, they might help you to figure this out:

```auto
PUT _ingest/pipeline/metricbeat_reindex
{
  "processors": [
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "metricbeat-6.8.0-",
        "date_rounding": "d",
        "index_name_format": "yyyy.MM.dd"
      }
    }
    ]
}

```

This adds an ingest pipeline that handles the index renaming, using a date format. You would need to use a different date\_rounding and/or index\_name\_format I guess. The documentation can be found here: [Date Index Name Processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-index-name-processor.html)

After that you can use this reindex to grab all your indices (notice the `-*` at the end of the source index name) and pipe them through the index renaming pipeline:

```auto
POST _reindex
{
  "source": {
    "index": "metricbeat-6.8.0-*"
  },
  "dest": {
    "pipeline": "metricbeat_reindex",
    "index": "its-md_dsx_metricbeat-6.8.0-dummy"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Praneet\_Khandelwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praneet_khandelwal/32/40565_2.png) [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\
**Post date:** [August 7, 2019, 8:59am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/3 "2019-08-07T08:59:18Z")

</div>

Thanks that helps

---

<div class="post-metadata">

**Author:** ![Praneet\_Khandelwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praneet_khandelwal/32/40565_2.png) [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\
**Post date:** [August 7, 2019, 2:58pm UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/4 "2019-08-07T14:58:09Z")

</div>

@BennyInc I was testing using the above method suggested by you. Seems to me there is something wrong in the date format conversion.  
Here is the pipeline I used

```auto
PUT _ingest/pipeline/temp-pipeline
{
  "description": "Test Pipeline",
  "processors" : [
    {
      "date_index_name" : {
       
        "field" : "date1",
        "index_name_prefix" : "myindex-",
        "date_rounding" : "s",
        "index_name_format": "YYYY-ww"
      }
    }
  ]
} 

```

And the doc example

```auto
PUT /myindex/_doc/1?pipeline=temp-pipeline
{
  "date1" : "2016-04-25T12:02:01.789Z",
  "tags": ["application:myapp", "env:Stage"]  
}

```

This is the output I get

```auto
{
  "_index" : "myindex-2015-17",
  "_type" : "_doc",
  "_id" : "11",
  "_version" : 1,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 3,
  "_primary_term" : 1
}

```

I can't understand why its outputting weak based year as 2015, that too in April, which does not lie on the boundary.  
Also If you try the "date1" as "2019-12-31T12:02:01.789Z", it gives "2019-01" as the output.

Can you help me with this.

The documentation for date time format is this as mentioned in the docs: [https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html)

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [August 8, 2019, 8:09am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/5 "2019-08-08T08:09:26Z")

</div>

For some strange reason your sample doesn't work on my Elastic 7.3 when using index\_name\_format `YYYY-ww` -- it does work with `yyyy-w`w however.  
Using that fix, I got `myindex-2016-01` for your sample from 2016-05-25.  
I tried setting the `date_rounding` to `w` instead, but that resulted in `myindex-2015-01`.

---

<div class="post-metadata">

**Author:** ![Praneet\_Khandelwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praneet_khandelwal/32/40565_2.png) [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\
**Post date:** [August 8, 2019, 8:51am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/6 "2019-08-08T08:51:36Z")

</div>

For 2016-05-25, even if we use "yyyy-ww", it should be "2016-17" and not "2016-01", right ?

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [August 8, 2019, 10:18am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/7 "2019-08-08T10:18:55Z")

</div>

That would be my understanding as well, yes.

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [August 8, 2019, 11:32am UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/9 "2019-08-08T11:32:13Z")

</div>

One more test, maybe someone from Elastic can comment on this:

```auto
PUT _ingest/pipeline/temp-pipeline
{
  "description": "Test Pipeline",
  "processors": [
    {
      "date_index_name": {
        "field": "date1",
        "index_name_prefix": "myindex-",
        "date_rounding": "w",
        "index_name_format": "yyyy-ww"
      }
    }
  ]
}

```

```auto
POST _ingest/pipeline/temp-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "date1": "2016-04-25T12:02:01.789Z",
        "tags": [
          "application:myapp",
          "env:Stage"
        ]
      }
    }
  ]
}

```

results in

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "<myindex-{2016-18||/w{yyyy-ww|UTC}}>",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "date1" : "2016-04-25T12:02:01.789Z",
          "tags" : [
            "application:myapp",
            "env:Stage"
          ]
        },
        "_ingest" : {
          "timestamp" : "2019-08-08T11:31:11.0199062Z"
        }
      }
    }
  ]
}

```

So the dynamic index name has the correct week in it already, but when we POST the same document directly, it gets applied incorrectly?

---

<div class="post-metadata">

**Author:** ![Praneet\_Khandelwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praneet_khandelwal/32/40565_2.png) [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\
**Post date:** [August 8, 2019, 12:01pm UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/10 "2019-08-08T12:01:23Z")

</div>

Yeah I had observed the same thing. And if you try for the year 2012 , 2011, it gives weird outputs. I thought maybe that had something to do with leap year but doesnt make sense to me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2019, 12:01pm UTC](https://discuss.elastic.co/t/reindex-daily-indices-to-weekly-indices/193780/11 "2019-09-05T12:01:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.