Reindex daily indices to weekly indices

Elastic Stack Elasticsearch
1

I am working on a reindexing problem in which I need to reindex the daily indices to weekly indices. I am using the reindex APi and trying to work with the script processor to make the changes. Can anyone help with what the script might look it?

2

I have done something similar lately. While my scripts won't directly solve your problem, they might help you to figure this out:

PUT _ingest/pipeline/metricbeat_reindex
{
  "processors": [
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "metricbeat-6.8.0-",
        "date_rounding": "d",
        "index_name_format": "yyyy.MM.dd"
      }
    }
    ]
}

This adds an ingest pipeline that handles the index renaming, using a date format. You would need to use a different date_rounding and/or index_name_format I guess. The documentation can be found here: Date Index Name Processor

After that you can use this reindex to grab all your indices (notice the -* at the end of the source index name) and pipe them through the index renaming pipeline:

POST _reindex
{
  "source": {
    "index": "metricbeat-6.8.0-*"
  },
  "dest": {
    "pipeline": "metricbeat_reindex",
    "index": "its-md_dsx_metricbeat-6.8.0-dummy"
  }
}
3

Thanks that helps

4

@BennyInc I was testing using the above method suggested by you. Seems to me there is something wrong in the date format conversion.
Here is the pipeline I used

PUT _ingest/pipeline/temp-pipeline
{
  "description": "Test Pipeline",
  "processors" : [
    {
      "date_index_name" : {
       
        "field" : "date1",
        "index_name_prefix" : "myindex-",
        "date_rounding" : "s",
        "index_name_format": "YYYY-ww"
      }
    }
  ]
}

And the doc example

PUT /myindex/_doc/1?pipeline=temp-pipeline
{
  "date1" : "2016-04-25T12:02:01.789Z",
  "tags": ["application:myapp", "env:Stage"]  
}

This is the output I get

{
  "_index" : "myindex-2015-17",
  "_type" : "_doc",
  "_id" : "11",
  "_version" : 1,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 3,
  "_primary_term" : 1
}

I can't understand why its outputting weak based year as 2015, that too in April, which does not lie on the boundary.
Also If you try the "date1" as "2019-12-31T12:02:01.789Z", it gives "2019-01" as the output.

Can you help me with this.

The documentation for date time format is this as mentioned in the docs: https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

5

For some strange reason your sample doesn't work on my Elastic 7.3 when using index_name_format YYYY-ww -- it does work with yyyy-ww however.
Using that fix, I got myindex-2016-01 for your sample from 2016-05-25.
I tried setting the date_rounding to w instead, but that resulted in myindex-2015-01.

6

For 2016-05-25, even if we use "yyyy-ww", it should be "2016-17" and not "2016-01", right ?

7

That would be my understanding as well, yes.

9

One more test, maybe someone from Elastic can comment on this:

PUT _ingest/pipeline/temp-pipeline
{
  "description": "Test Pipeline",
  "processors": [
    {
      "date_index_name": {
        "field": "date1",
        "index_name_prefix": "myindex-",
        "date_rounding": "w",
        "index_name_format": "yyyy-ww"
      }
    }
  ]
}

POST _ingest/pipeline/temp-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "date1": "2016-04-25T12:02:01.789Z",
        "tags": [
          "application:myapp",
          "env:Stage"
        ]
      }
    }
  ]
}

results in

{
  "docs" : [
    {
      "doc" : {
        "_index" : "<myindex-{2016-18||/w{yyyy-ww|UTC}}>",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "date1" : "2016-04-25T12:02:01.789Z",
          "tags" : [
            "application:myapp",
            "env:Stage"
          ]
        },
        "_ingest" : {
          "timestamp" : "2019-08-08T11:31:11.0199062Z"
        }
      }
    }
  ]
}

So the dynamic index name has the correct week in it already, but when we POST the same document directly, it gets applied incorrectly?

10

Yeah I had observed the same thing. And if you try for the year 2012 , 2011, it gives weird outputs. I thought maybe that had something to do with leap year but doesnt make sense to me.