How to add time of ingestion to the document?

Elastic Stack Elasticsearch
1

Hi Elasticians,
I've created pipeline pipeline_add_ingest_timestamp. I would like to achieve add time of ingesting by elasticsearch to the document using this pipeline. What am I doing wrong? The field is not populate to the document using this pipeline:

PUT _ingest/pipeline/pipeline_add_ingest_timestamp
{ 
  "description": "Adds event.ingested field which represents time of ingestion.",
  "processors": [
    {
      "set": {
        "field": "event.ingested",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}

 

GET _ingest/pipeline

 

  "pipeline_add_ingest_timestamp" : {
    "description" : "Adds event.ingested field which represents time of ingestion.",
    "processors" : [
      {
        "set" : {
          "field" : "event.ingested",
          "value" : "{{_ingest.timestamp}}"
        }
      }
    ]
  }
1 Like
2

Your ingest pipeline looks good, I copied it to my own Kibana and used the Simulate pipeline API to test it - and it resulted in an event.ingested field with a timestamp in each document.

So I assume your problem is how to trigger the pipeline when you index new documents? For that you need to supply the pipeline-parameter with the name of your pipeline. Let's say you name it "my_timestamp_pipeline" then you need to index new documents like this:

PUT my-index/_doc/my-id?pipeline=my_timestamp_pipeline
{
  "foo": "bar"
}

If you use a specific programming language on the client side (Perl, PHP, Python etc) you'll need to look up the documentation for that language to learn how to add the pipeline-parameter with each indexing request.

For more info see the Ingest node documentation.

1 Like
3

Thank you @Bernt_Rostad. I've had missing parameter pipeline in Logstash configuration of logstash-output-elasticsearch plugin. Now it works.

Thank you!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.