# Data Correction

**URL:** <https://discuss.elastic.co/t/data-correction/260256>\
**Category:** Elasticsearch\
**Created:** [January 5, 2021, 9:01pm UTC](https://discuss.elastic.co/t/data-correction/260256 "2021-01-05T21:01:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![stevezemlicka](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\
**Post date:** [January 5, 2021, 9:01pm UTC](https://discuss.elastic.co/t/data-correction/260256/1 "2021-01-05T21:01:18Z")

</div>

I have an index that is comprised of two separate data dumps. Upon further analysis, it was discovered that one of those seems to have had a 1hr shift. This has been partially rectified by using offsets with the visualization tools but I'd like to correct that on the backend.

I can modify the CSV and re-ingest but I'm wondering what it looks like to shift that data using elasticsearch. I suppose I write a GET query to match the documents. But after that is the update performed with a POST command or something similar?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 5, 2021, 9:03pm UTC](https://discuss.elastic.co/t/data-correction/260256/2 "2021-01-05T21:03:06Z")

</div>

You can do an in place reindex, the concept is the same as this, just ignore the upgrade parts - [https://www.elastic.co/guide/en/elasticsearch/reference/7.10/reindex-upgrade-inplace.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/reindex-upgrade-inplace.html) 🙂

---

<div class="post-metadata">

**Author:** ![stevezemlicka](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\
**Post date:** [January 11, 2021, 9:39pm UTC](https://discuss.elastic.co/t/data-correction/260256/3 "2021-01-11T21:39:52Z")

</div>

Sorry for the delay, I'm finally getting back around to this. I checked that out but I don't seem to be able to put it together. I don't think i explained my issue very well.

I have an index where the data is accurate from July 2019 though November 3rd 2019. But from November 3rd 2019 through August 2020, the data is shifted by +1hr. Is there a way for me run an elasticsearch command to shift all the @timestamps for documents in that index for that timerange by -1hr?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 11, 2021, 9:42pm UTC](https://discuss.elastic.co/t/data-correction/260256/4 "2021-01-11T21:42:23Z")

</div>

Yes, you need to reindex and change that timestamp in the reindex process 🙂

---

<div class="post-metadata">

**Author:** ![stevezemlicka](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\
**Post date:** [January 12, 2021, 3:48pm UTC](https://discuss.elastic.co/t/data-correction/260256/5 "2021-01-12T15:48:37Z")

</div>

I think I'm missing something and it's probably due to the fact that I'm still pretty noob with elasticsearch. I think what I need help with is the following excerpt from the document you linked:

> You can use a script to perform any necessary modifications to the document data and metadata during reindexing.

From what I gather, I can use something like the following to reindex:

```auto
POST _reindex
{
   "source":{
      "index":"source",
      "query": {
        "match": {
           "field_name": "text"
         }
      }
   },
   "dest":{
      "index":"destination"
   }
}

```

But I need to put something more in the destination to transform the timestamp. Can I simply use offset=-1h with the destination to perform that transformation? For example:

```auto
   "dest":{
      "index":"destination",
      "offset=-1h"
   }

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 12, 2021, 11:36pm UTC](https://discuss.elastic.co/t/data-correction/260256/6 "2021-01-12T23:36:06Z")

</div>

Not sure where you got that offset from, but I don't believe it's valid.

You will probably want to add a script to do some date math like this;

```auto
{
  "script_fields": {
    "new_date_field": {
      "script": {
        "inline": "doc['date_field'].value + 3600"
      }
    }
  }
}

```

Where 3600 is the number of seconds in an hour to add onto your existing `date_field` (or whatever it's called).

---

<div class="post-metadata">

**Author:** ![stevezemlicka](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\
**Post date:** [January 13, 2021, 4:55pm UTC](https://discuss.elastic.co/t/data-correction/260256/7 "2021-01-13T16:55:26Z")

</div>

I got the offset because I have no clue what I'm doing with this (or rather I'm still learning) and since I've been using Timelion, I figured I'd demonstrate that I was trying rather than just seeming like I was leeching.

Thank you, I think that's exactly what I need. Further investigation has revealed that I think the problem is with some devices in this dataset not accounting for DST...so this will start to get tricky, but I think with the info you provided, I should be able to figure it out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 10, 2021, 4:55pm UTC](https://discuss.elastic.co/t/data-correction/260256/8 "2021-02-10T16:55:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.