Reindex daily indices to weekly indices

Elastic Stack Elasticsearch
1

I am working on a reindexing problem in which I need to reindex the daily indices to weekly indices. I am using the reindex APi and trying to work with the script processor to make the changes. Can anyone help with what the script might look it?

1 Like
2

I have done something similar lately. While my scripts won't directly solve your problem, they might help you to figure this out:

PUT _ingest/pipeline/metricbeat_reindex
{
  "processors": [
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "metricbeat-6.8.0-",
        "date_rounding": "d",
        "index_name_format": "yyyy.MM.dd"
      }
    }
    ]
}

This adds an ingest pipeline that handles the index renaming, using a date format. You would need to use a different date_rounding and/or index_name_format I guess. The documentation can be found here: Date Index Name Processor

After that you can use this reindex to grab all your indices (notice the -* at the end of the source index name) and pipe them through the index renaming pipeline:

POST _reindex
{
  "source": {
    "index": "metricbeat-6.8.0-*"
  },
  "dest": {
    "pipeline": "metricbeat_reindex",
    "index": "its-md_dsx_metricbeat-6.8.0-dummy"
  }
}
3

Thanks that helps

4

@BennyInc I was testing using the above method suggested by you. Seems to me there is something wrong in the date format conversion.
Here is the pipeline I used

PUT _ingest/pipeline/temp-pipeline
{
  "description": "Test Pipeline",
  "processors" : [
    {
      "date_index_name" : {
       
        "field" : "date1",
        "index_name_prefix" : "myindex-",
        "date_rounding" : "s",
        "index_name_format": "YYYY-ww"
      }
    }
  ]
}

And the doc example

PUT /myindex/_doc/1?pipeline=temp-pipeline
{
  "date1" : "2016-04-25T12:02:01.789Z",
  "tags": ["application:myapp", "env:Stage"]  
}

This is the output I get

{
  "_index" : "myindex-2015-17",
  "_type" : "_doc",
  "_id" : "11",
  "_version" : 1,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 3,
  "_primary_term" : 1
}

I can't understand why its outputting weak based year as 2015, that too in April, which does not lie on the boundary.
Also If you try the "date1" as "2019-12-31T12:02:01.789Z", it gives "2019-01" as the output.

Can you help me with this.

The documentation for date time format is this as mentioned in the docs: https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

5

For some strange reason your sample doesn't work on my Elastic 7.3 when using index_name_format YYYY-ww -- it does work with yyyy-ww however.
Using that fix, I got myindex-2016-01 for your sample from 2016-05-25.
I tried setting the date_rounding to w instead, but that resulted in myindex-2015-01.

6

For 2016-05-25, even if we use "yyyy-ww", it should be "2016-17" and not "2016-01", right ?

1 Like
7

That would be my understanding as well, yes.

1 Like
9

One more test, maybe someone from Elastic can comment on this:

PUT _ingest/pipeline/temp-pipeline
{
  "description": "Test Pipeline",
  "processors": [
    {
      "date_index_name": {
        "field": "date1",
        "index_name_prefix": "myindex-",
        "date_rounding": "w",
        "index_name_format": "yyyy-ww"
      }
    }
  ]
}

POST _ingest/pipeline/temp-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "date1": "2016-04-25T12:02:01.789Z",
        "tags": [
          "application:myapp",
          "env:Stage"
        ]
      }
    }
  ]
}

results in

{
  "docs" : [
    {
      "doc" : {
        "_index" : "<myindex-{2016-18||/w{yyyy-ww|UTC}}>",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "date1" : "2016-04-25T12:02:01.789Z",
          "tags" : [
            "application:myapp",
            "env:Stage"
          ]
        },
        "_ingest" : {
          "timestamp" : "2019-08-08T11:31:11.0199062Z"
        }
      }
    }
  ]
}

So the dynamic index name has the correct week in it already, but when we POST the same document directly, it gets applied incorrectly?

1 Like
10

Yeah I had observed the same thing. And if you try for the year 2012 , 2011, it gives weird outputs. I thought maybe that had something to do with leap year but doesnt make sense to me.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.