# Fetch and store data from an index every hour

**URL:** <https://discuss.elastic.co/t/fetch-and-store-data-from-an-index-every-hour/278332>\
**Category:** Elasticsearch\
**Created:** [July 10, 2021, 1:25pm UTC](https://discuss.elastic.co/t/fetch-and-store-data-from-an-index-every-hour/278332 "2021-07-10T13:25:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrsac7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrsac7/32/91460_2.png) [@mrsac7](https://discuss.elastic.co/u/mrsac7)\
**Post date:** [July 10, 2021, 1:25pm UTC](https://discuss.elastic.co/t/fetch-and-store-data-from-an-index-every-hour/278332/1 "2021-07-10T13:25:45Z")

</div>

Dear community,

I want to create a script that can -

- Fetch data from a given **source index**
- Process the data and store into a **destination index**

The script will be scheduled to run once every hour.

The data contains **timestamp** field. My approach is to keep record of "fetched\_timestamp" variable which indicates that we need to fetch documents after this timestamp. To implement this, I'm using scroll api with the following query -

```auto
"query": {
    "range": {
        "@timestamp": {
            "gte": timestamp
        }
    }
}

```

This will create a scroll context and with the help of scroll\_id, I can fetch all the records from the current snapshot of the source index. After the current snapshot is scrolled out, I will update the "fetched\_timestamp" variable to the max timestamp among the documents fetched and will get the next batch of documents.

The problem is that new documents will continue to be added to the source index all day long. It may happen that a document with an older timestamp (back-fill) is added after a newer timestamp in the source index. Since, I am always fetching documents after a certain timestamp therefore I can lose the documents which are inserted recently but have older timestamps.

Is there any way to get all the documents? Any suggestion is very much appreciated.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 12, 2021, 8:21am UTC](https://discuss.elastic.co/t/fetch-and-store-data-from-an-index-every-hour/278332/2 "2021-07-12T08:21:06Z")

</div>

So, apparently the timestamp of the log creation is not the correct one. Maybe the timestamp of the ingestion into Elasticsearch could be more helpful. See the `_ingest` timestamp at [Ingest pipelines | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/ingest.html)

Maybe something like [transforms](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/transforms.html) can help you out, so you don't have to come up with your own logic. Can you explain your use-case in a bit more detail?

---

<div class="post-metadata">

**Author:** ![mrsac7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrsac7/32/91460_2.png) [@mrsac7](https://discuss.elastic.co/u/mrsac7)\
**Post date:** [July 13, 2021, 7:11am UTC](https://discuss.elastic.co/t/fetch-and-store-data-from-an-index-every-hour/278332/3 "2021-07-13T07:11:01Z")

</div>

I think you understood it correctly. There are two timestamp fields -

1. **request\_timestamp** : timestamp at which request was made.
2. **created\_timestamp** : timestamp at which document was ingested.

Earlier, I was trying to use the **request\_timestamp** due to which the issue was raised. I think using **created\_timestamp** is a better approach.

Thanks for the suggestion.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 10, 2021, 7:11am UTC](https://discuss.elastic.co/t/fetch-and-store-data-from-an-index-every-hour/278332/4 "2021-08-10T07:11:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.