# How to know the time when elasticsearch indexed the data

**URL:** <https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603>\
**Category:** Elasticsearch\
**Created:** [December 14, 2020, 3:46pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603 "2020-12-14T15:46:18Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![soufian.eldouqe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/soufian.eldouqe/32/42261_2.png) [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\
**Post date:** [December 14, 2020, 3:46pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/1 "2020-12-14T15:46:18Z")

</div>

Hello,

is there any way to know the time that elasticsearch indexed the data ? in others words, when did the index received de data.

Thank you

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 15, 2020, 4:21am UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/2 "2020-12-15T04:21:01Z")

</div>

No. It must be something you add to the data itself.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 15, 2020, 4:23am UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/3 "2020-12-15T04:23:34Z")

</div>

You can use an ingest pipeline to [add a timestamp to each document it at indexing time](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/accessing-data-in-pipelines.html#accessing-ingest-metadata).

---

<div class="post-metadata">

**Author:** ![soufian.eldouqe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/soufian.eldouqe/32/42261_2.png) [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\
**Post date:** [December 15, 2020, 3:53pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/4 "2020-12-15T15:53:33Z")

</div>

Thank you for the answer.

So I have to create another pipeline to access the metadata of my pipeline. Is it correct? if so, can you give me an exemple of the conf of this pipeline please?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 15, 2020, 4:09pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/5 "2020-12-15T16:09:10Z")

</div>

The link @Christian_Dahlqvist shared gives a concrete example.

Create a pipeline

```auto
PUT _ingest/pipeline/add_date
{
  "description" : "This pipeline add the ingest time to the document",
  "processors" : [
    {
      "set" : {
        "field": "received",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}

```

Index a document

```auto
PUT index/_doc/1?pipeline=add_date
{
  "foo": "bar"
}

```

Get back the document

```auto
GET index/_doc/1

```

You should see:

```auto
{
  "foo": "bar",
  "received": "THE INGEST DATE HERE" 
}

```

---

<div class="post-metadata">

**Author:** ![soufian.eldouqe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/soufian.eldouqe/32/42261_2.png) [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\
**Post date:** [December 15, 2020, 4:25pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/6 "2020-12-15T16:25:38Z")

</div>

Thank you so much.

Just one last question please. I see that with this conf it will be applied to only one document ( id = 1 ). how can I have that applied to alla my documents and the future document that will be ingested automatically in my index ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 15, 2020, 4:28pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/7 "2020-12-15T16:28:23Z")

</div>

You define a default ingest pipeline associated to an index. See

> **[Index modules | Elasticsearch Guide \[7.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index-modules.html#dynamic-index-settings)**

It says:

> `index.default_pipeline`
> 
> The default [ingest node](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/ingest.html) pipeline for this index. Index requests will fail if the default pipeline is set and the pipeline does not exist. The default may be overridden using the `pipeline` parameter. The special pipeline name `_none` indicates no ingest pipeline should be run.
> 
> `index.final_pipeline`
> 
> The final [ingest node](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/ingest.html) pipeline for this index. Index requests will fail if the final pipeline is set and the pipeline does not exist. The final pipeline always runs after the request pipeline (if specified) and the default pipeline (if it exists). The special pipeline name `_none` indicates no ingest pipeline will run.

---

<div class="post-metadata">

**Author:** ![soufian.eldouqe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/soufian.eldouqe/32/42261_2.png) [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\
**Post date:** [December 15, 2020, 4:31pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/8 "2020-12-15T16:31:27Z")

</div>

thank you 🙏 🙏

---

<div class="post-metadata">

**Author:** ![soufian.eldouqe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/soufian.eldouqe/32/42261_2.png) [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\
**Post date:** [December 21, 2020, 4:34pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/9 "2020-12-21T16:34:21Z")

</div>

I'm sorry to come boder you once again about this subject. Every thing works fine.

There is only one issue :  
I have a rollover system working on my indices. when the rollover is triggered, there will be a new index created `ix_xxx-2020.12.21-000002`.

The issue is my index will not have the defaut config of the ingest pipeline. so I should re-execute the config eachtime I have a new rollover index.

Is there a way to automate this and avoid following error (when I consult my data in the discover ) :

```
ix_xxx-2020.12.21-000002 node yyyyyy
Type script_exception
Reason runtime error
Script stack
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)
if (doc['received'].size() == 0) 
    ^---- HERE
Script
if (doc['received'].size() == 0) return 0;
(doc['received'].value.millis - doc['@timestamp'].value.millis) / 1000 ;

Lang painless
Caused by type illegal_argument_exception
Caused by reason No field found for [received] in mapping with types []

```

The _ **expression\_field** _ above calculate the duration between FileBeat `("doc['@timestamp']")` and the ingestion of data in the index `("doc['received']")` and this works fine for the first index `ix_xxx-2020.12.21-000001` before the rollover system

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 21, 2020, 5:31pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/11 "2020-12-21T17:31:15Z")

</div>

> [@soufian.eldouqe](#):
>
> The issue is my index will not have the defaut config of the ingest pipeline. so I should re-execute the config eachtime I have a new rollover index.

Add the required settings to an index template and it should automatically apply to all new indices.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2021, 5:31pm UTC](https://discuss.elastic.co/t/how-to-know-the-time-when-elasticsearch-indexed-the-data/258603/12 "2021-01-18T17:31:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.