How to automatically set the server date/time when adding data?

vmassol · February 9, 2022, 10:28am

Hi there,

In old ES versions, I was using the _timestamp field but it's been removed and it's now recommended to set the date on the client side. However, I'm surprised to not find any easy way to continue to set it on the server side. Setting the date on the client side is risky for our use case since we don't control the clients (they can be anywhere in the world) and they can put wrong date or in the wrong timezone. Our use case is to record pings from clients and I'd much prefer to set the date using the ES server date/time.

Is there an easy way to do this?

Thanks a lot
-Vincent

dadoonet · February 9, 2022, 10:49am

Bonjour Vincent!

I believe you can do something like this using an ingest pipeline with a Set processor.

And you can make the pipeline a default pipeline for your index. (See Ingest pipelines | Elasticsearch Guide [7.17] | Elastic)

Full example (from this example Ingest pipelines | Elasticsearch Guide [7.17] | Elastic):

PUT _ingest/pipeline/set-timestamp
{
  "description": "sets the timestamp",
  "processors": [
    {
      "set": {
        "field": "timestamp",
        "value": "{{{_ingest.timestamp}}}"
      }
    }
  ]
}
DELETE test
PUT test
{
  "settings": {
    "index.default_pipeline": "set-timestamp"
  },
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}
PUT test/_doc/1
{
  "foo": "bar"
}
GET test/_doc/1

Gives:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "foo" : "bar",
    "timestamp" : "2022-02-09T10:47:34.880561Z"
  }
}

vmassol · February 9, 2022, 12:38pm

hey, hi David!

Thanks a lot for your answer.

I should have mentioned that I saw this answer in some answer on stackoverflow but I thought it looked a bit complex, hence why I asked the question here. So I gather that there's no simpler way and I'll go in this direction, thanks again!

PS: The concept of pipelines is pretty cool!

vmassol · February 9, 2022, 12:45pm

@dadoonet Sorry, another question related to this. Do you know if this ingest pipeline will overwrite any value set in the timestamp field from your example? Is it possible to say to replace the field value only if it's empty or not set?

Thanks

vmassol · February 9, 2022, 12:51pm

To be more precise, I'd like to set 2 dates:

pingDate: always set it with an ingest pipeline to be the current server date
firstPingDate: I do some query to find the first ping entry's firstPingDate for a given instance id. If it exists, I set it to that value. If it doesn't exist, I'd like to set it to the current server date. I'd like it to be the same value as the pingDate too.

So 3 questions:

Do I need 2 pipelines?
How can I set firstPingDate in a pipeline, only if not set already?
How can I be sure to use the same value as the pingDate provided by the ES server?

Lots of questions, sorry...

Thanks!

vmassol · February 9, 2022, 12:53pm

Actually I'm using the wrong terminology. I need 2 processors I think.

vmassol · February 9, 2022, 12:53pm

Found the answer with the override config param in Set processor | Elasticsearch Guide [7.17] | Elastic

Tomo_M · February 9, 2022, 12:54pm

Af far as you are using Update API, override parameter or if paramter in set processors can detect whether there is existing value or not.

But if you use Index API, both paramters does not work.

vmassol · February 9, 2022, 12:55pm

Seems I coud use the copy_from config param from Set processor | Elasticsearch Guide [7.17] | Elastic

vmassol · February 10, 2022, 10:07am

@dadoonet hmm I can't find aJava API to use copy_from, it seems to be missing. This is what I'm trying:

        client.ingest().putPipeline(b0 -> b0
            .id("set-timestamp")
            .description("Set pingDate to be the server date/time and fill firstPingDate if empty")
            .processors(b1 -> b1
                .set(b2 -> b2
                    .field(PROPERTY_PING_DATE)
                    .value(JsonData.of("{{{_ingest.timestamp}}}")))
                .set(b3 -> b3
                    .override(false)
                    .field(PROPERTY_FIRST_PING_DATE)
                    .copy_from(PROPERTY_PING_DATE)
                )));

BTW the second set() isn't correct syntactically. How do I setup 2 "set" processors using the java API?

Thx!

vmassol · February 10, 2022, 10:10am

For the second question re multiple processors, it seems the API is not finished but I've found a workaround with:

        Processor processor1 = Processor.of(b0 -> b0
            .set(b1 -> b1
                .field(PROPERTY_PING_DATE)
                .value(JsonData.of("{{{_ingest.timestamp}}}"))));
        Processor processor2 = Processor.of(b0 -> b0
            .set(b3 -> b3
                .override(false)
                .field(PROPERTY_FIRST_PING_DATE)
                .copy_from(PROPERTY_PING_DATE)));
        client.ingest().putPipeline(b0 -> b0
            .id("set-timestamp")
            .description("Set pingDate to be the server date/time and fill firstPingDate if empty")
            .processors(processor1, processor2);

Note that copy_from() is not working as it doesn't exist. So I still don't know how to achieve that part.

Thx

dadoonet · February 10, 2022, 11:04am

Bringing in @swallez as this part is the one he is working on.

dadoonet · February 10, 2022, 11:06am

That said, why not using value in that case?

Like "value": "{{{timestamp}}}"

vmassol · February 10, 2022, 2:03pm

Yes that works fine, as a workaround (I guess it would be cleaner to use copy_from since it's meant for that)

Thanks!

For future readers, this is what I got to:

        Processor processor1 = Processor.of(b0 -> b0
            .set(b1 -> b1
                .field(PROPERTY_PING_DATE)
                .value(JsonData.of("{{{_ingest.timestamp}}}"))));
        Processor processor2 = Processor.of(b0 -> b0
            .set(b3 -> b3
                .override(false)
                .field(PROPERTY_FIRST_PING_DATE)
                .value(JsonData.of(String.format("{{{%s}}}", PROPERTY_PING_DATE)))));
        client.ingest().putPipeline(b0 -> b0
            .id(PIPELINE_NAME)
            .description("Set pingDate to be the server date/time and fill firstPingDate if empty")
            .processors(processor1, processor2));

swallez · February 10, 2022, 4:46pm

Thank @dadoonet. It's actually missing in the API specification and I opened an issue to fix this.

github.com/elastic/elasticsearch-specification

Missing fields in SetProcessor

opened 04:45PM - 10 Feb 22 UTC

swallez

missing type backport 8.0 backport 7.17 backport 8.1 lang:java

The ingest `set` processor ([type SetProcessor](https://github.com/elastic/elast…icsearch-specification/blob/025d85d754f8e91a9bdc2409631a66f87340096f/specification/ingest/_types/Processors.ts#L305)) is missing a lot of fields [described in the docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html). Reported in [this Discuss topic](https://discuss.elastic.co/t/how-to-automatically-set-the-server-date-time-when-adding-data/296708/10) related to using `copy_to` in the Java Client

dadoonet · February 10, 2022, 6:14pm

To me copy_from is helpful when you want to copy a full structure of fields, like an object.
For a single field, I think that value is the way to go.

system · March 10, 2022, 6:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Set default time field on ingest_timestamp Elasticsearch ingest-pipeline	1	393	February 2, 2021
Default timestamp in date Field Elasticsearch	5	3377	April 12, 2018
_ingest.timestamp does not match my local time Elasticsearch ingest-pipeline	4	393	December 5, 2023
Automatically add current timestamp when indexing Elasticsearch	4	465	July 9, 2019
Dec 12th, 2018: [EN][Elasticsearch] Automatically adding a timestamp to documents Advent Calendar	1	8428	December 1, 2019

How to automatically set the server date/time when adding data?

Related topics