Best practice for integrating automated created_at & updated_at fields in Elasticsearch 8.4

Elastic Stack Elasticsearch
1

Hello, this is my first transition from other DB (SQL / NOSQL based) to Elasticsearch.
I am having troubles choosing the best practiced methods of integrating created_at & updated_at fields.
I am familiar with ingest_pipelines for adding created_at fields while indexing a document, however I am aware that Update API do not support ingest_pipelines yet. So in order to circumvent this problem, we can use scripted_upsert to modify updated_at field. But I am not sure if this is the most recommended way to integrate created_at & updated_at in the DB. Could you share some ideas on how to store and modify updated_at & created_at ? TIA.

3

Welcome!

The best practice is to control that from the source application which actually creates the document.

What is the source application? Could you tell more about your use case?

In general, I never use the update API but only the Index API. Why do you need the update API here?

4

@dadoonet

  1. i use Index API to index a new document, and Update API to modify an existing document.
  2. My source application is just a group of REST API who is in charge of performing normal CRUD operations.

The best practice is to control that from the source application which actually creates the document.

by this, do you mean I just append a created_at / updated_at field to the JSON before indexing a new document or updating an existing one?

5

Why not using the index API when you want to modify the document?

I mean that behind the scene, the Update API is just:

  • GET the old document
  • Merge it with the new one
  • Index this full document

Yes.

6

@dadoonet
Thank you for your reply.
I think I am going to use Index API instead of Update API.
So now, do you think I should control created_at updated_at from source application or just write an ingest_pipeline for that, as I dont have any dependency on Update API?

Sorry to make this post long, will be closing this after this reply. Thank you for helping me out!

7

IMHO it's always better to do most of the things on the app side. So elasticsearch has less to do and it's easier for you to maintain.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.