Live index data transformation by using ingest pipeline

Elastic Stack Elasticsearch
1

Dear Team,

I'm new to ELK stack and need your support on this,

I had an index where live date in coming form the kafka.
Recently I had converted string data to date format by using the ingest pipeline.
After that I had created reindexing to the new index, but here the problem started,
After creating new Index the date field was showing (new field created by converting string to date in ingest pipeline).
I want to know that if the new date coming to this index will be transformed or should we need to do any changes in order to reflect the new data in replicated index.

Thanks and Regards,
Pavan Satya.

2

Welcome!

Could you illustrate this with an example of before vs after?

3

Hi Dadoonet,

We are having data in the following Fields:
lastModifiedDate: 2024-05-20 18:49:35.236

By using Ingest pipeline we had transformed it as followed in a new field.
last_Modified_Date:
May 20, 2024 @ 21:49:35.236

image
image1003×198 21.9 KB

After Doing this change, we had reindexed the old index as followed. (by giving ingest pipeline).

POST /_reindex
{
  "source": {
    "index": "testforresponsetimestringtodate"
  },
  "dest": {
    "index": "testforresponsetimestringtodate2",
    "pipeline": "testing_for_string2Date"
  }
}

Here I need to know that. For original Index (testforresponsetimestringtodate)
we are getting live data from the servers. So we want this data need to be reflected in new index (testforresponsetimestringtodate2) with date transformation.

Hope this will give you better understanding on this.

  1. If we give default pipeline for the original index, then it will give transformed data (date transformation) for the new upcoming data?
    if there is any other solution. kindly let me know.

Thanks and Regards,
Pavan.

4

Why not writing the data to testforresponsetimestringtodate2 then? Instead of writing in the first one, then reading, then reindex everything a second time?

5

Hi David,

I didn't get you,
Could you please clarify me more on this.

Thanks and Regards,
Pavan.

6

Let me put this clearly,

Transformed the data (date from "2024-05-20 18:48:16.063" to "May 20, 2024 @ 21:48:16.063"). So, in order to do that I had used data ingest pipeline (date transformation with creating a new field).
After that I had reindexed it with giving default pipeline as " ingest pipeline " which I created.
My ultimate goal is to transform the new data that is coming to the index (either it may be new index or old one).
I want to see the new data in this format "May 20, 2024 @ 21:48:16.063" (which I did in ingest pipeline) that need to be shown continually in the index field. Kindly help me here how to do that.

7

It's "just" a formatting issue?

What does your pipeline look like?

8

Hi David,

This is the pipeline transformation we used.

[
  {
    "date": {
      "field": "lastModifiedDate",
      "formats": [
        "yyyy-MM-dd HH:mm:ss.SSS",
        "ISO8601"
      ],
      "target_field": "last_updatedtime",
      "output_format": "MM DD, yyyy @ HH:mm:ss.SSS"
    }
  },
  {
    "date": {
      "field": "createdDate",
      "formats": [
        "yyyy-MM-dd'T'HH:mm:ss.SSSXXX",
        "yyyy-MM-ddTHH:mm:ss.SSSXXX",
        "ISO8601"
      ],
      "target_field": "Ordercreatedtime",
      "output_format": "MM DD, yyyy @ HH:mm:ss.SSS"
    }
  }
]

We are getting data from Elasticsearch Service Sink Connector and we want to transform the date's in to correct format with new field names. It was working fine when I tested it with sample data. Now we want to get transformed date for the new upcoming data in the same Index.

Help me in this.
From the code "lastModifiedDate" and "createdDate" are the raw data that we are getting. In pipeline we transformed them by giving new field names "Ordercreatedtime" and "last_updatedtime".

We want this new fields need to be reflected in index.

Thanks and Regards,
Pavan.