Ingestion pipeline: setting @timestamp

Elastic Stack Elastic Agent
1

Hi,

I'd like to set the @timestamp field to the value of event.ingested. It doesn't seem to be possible. My ingestion pipeline:

{
  "processors": [
    {
      "set": {
        "ignore_empty_value": true,
        "field": "time.original",
        "copy_from": "@timestamp"
      }
    },
    {
      "set": {
        "ignore_empty_value": true,
        "field": "@timestamp",
        "copy_from": "event.ingested"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
      }
    }
  ]
}

In events the value of @timestamp remains unchanged

Thanks for your help!

2

Hello,

Where are you using this? On a custom pipeline for some Elastic Agent integration?

3

Can you give the steps to reproduce the problem? I just tried that pipeline using this:

PUT test/_doc/1?pipeline=test-pipeline
{
    "event": {"ingested": 1234},
    "@timestamp": 5679
}

I get the expected result:

{
  "_index": "test",
  "_id": "1",
  "_version": 17,
  "_seq_no": 17,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "@timestamp": 1234,
    "time": {
      "original": 5679
    },
    "event": {
      "ingested": 1234
    }
  }
}
1 Like
4

The main issue here is that if the OP is using a custom pipeline from any Elastic Agent integration, the event.ingested will not exist at the time the custom ingest pipeline is executed.

The event.ingested field is created by the .fleet_final_pipeline-1 ingest pipeline, which is set as the index.final_pipeline for all Elastic Agent integrations, so this will be only executed after all other pipelines.

All customizations when using Elastic Agent integrations will happen before the field is created, so the user cannot set the @timestamp to the value of event.ingested.

I think the close it can get is to use the value of _ingest.metadata, it will not be the same but probably very close, here is the example from the documentation.

2 Likes