How to know the time when elasticsearch indexed the data

Elastic Stack Elasticsearch
1

Hello,

is there any way to know the time that elasticsearch indexed the data ? in others words, when did the index received de data.

Thank you

2

No. It must be something you add to the data itself.

3

You can use an ingest pipeline to add a timestamp to each document it at indexing time.

4

Thank you for the answer.

So I have to create another pipeline to access the metadata of my pipeline. Is it correct? if so, can you give me an exemple of the conf of this pipeline please?

5

The link @Christian_Dahlqvist shared gives a concrete example.

Create a pipeline

PUT _ingest/pipeline/add_date
{
  "description" : "This pipeline add the ingest time to the document",
  "processors" : [
    {
      "set" : {
        "field": "received",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}

Index a document

PUT index/_doc/1?pipeline=add_date
{
  "foo": "bar"
}

Get back the document

GET index/_doc/1

You should see:

{
  "foo": "bar",
  "received": "THE INGEST DATE HERE" 
}
6

Thank you so much.

Just one last question please. I see that with this conf it will be applied to only one document ( id = 1 ). how can I have that applied to alla my documents and the future document that will be ingested automatically in my index ?

7

You define a default ingest pipeline associated to an index. See

It says:

index.default_pipeline

The default ingest node pipeline for this index. Index requests will fail if the default pipeline is set and the pipeline does not exist. The default may be overridden using the pipeline parameter. The special pipeline name _none indicates no ingest pipeline should be run.

index.final_pipeline

The final ingest node pipeline for this index. Index requests will fail if the final pipeline is set and the pipeline does not exist. The final pipeline always runs after the request pipeline (if specified) and the default pipeline (if it exists). The special pipeline name _none indicates no ingest pipeline will run.

8

thank you :pray: :pray:

9

I'm sorry to come boder you once again about this subject. Every thing works fine.

There is only one issue :
I have a rollover system working on my indices. when the rollover is triggered, there will be a new index created ix_xxx-2020.12.21-000002.

The issue is my index will not have the defaut config of the ingest pipeline. so I should re-execute the config eachtime I have a new rollover index.

Is there a way to automate this and avoid following error (when I consult my data in the discover ) :

ix_xxx-2020.12.21-000002 node yyyyyy
Type    script_exception
Reason    runtime error
Script stack
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)
if (doc['received'].size() == 0) 
    ^---- HERE
Script
if (doc['received'].size() == 0) return 0;
(doc['received'].value.millis - doc['@timestamp'].value.millis) / 1000 ;

Lang    painless
Caused by type    illegal_argument_exception
Caused by reason    No field found for [received] in mapping with types []

The expression_field above calculate the duration between FileBeat ("doc['@timestamp']") and the ingestion of data in the index ("doc['received']") and this works fine for the first index ix_xxx-2020.12.21-000001 before the rollover system

11

Add the required settings to an index template and it should automatically apply to all new indices.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.