Convert Timestamp to milliseconds in ingest pipeline

[Rakesh_Rayabarapu (Rakesh R)]

Hi,

I am trying to find a way to get millis from string date time field using ingest pipeline and getting following error.

please help me to correct the script.

POST /_ingest/pipeline/_simulate
{
"pipeline": {
"description": "_description",
"processors": [
{
"script": {
"source": "ctx.time_millis = ZonedDateTime.parse(ctx.TimeStamp.getMillis)"
}
}
]
},
"docs": [
{
"_index": "index",
"_id": "id",
"_source": {
"TimeStamp": "2022-01-13T10:42:10.982"
}
}
]
}

[Rakesh_Rayabarapu (Rakesh R)]

Error :
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"ctx.time_millis = ZonedDateTime.parse(ctx.TimeStamp.getMillis)",
" ^---- HERE"
],
"script" : "ctx.time_millis = ZonedDateTime.parse(ctx.TimeStamp.getMillis)",
"lang" : "painless",
"position" : {
"offset" : 51,
"start" : 0,
"end" : 62
},
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "dynamic getter [java.lang.String, getMillis] not found"
}
}
}

[majagrubic (Maja)]

I am not sure, but could it maybe be that you need ZonedDateTime.parse(ctx.TimeStamp.getMillis())? getMilis() as a method, opposed to a property?

[Rakesh_Rayabarapu (Rakesh R)]

Hello Maja, thank you for the response..

We actually looking to get this action painless script using ingest pipeline.

Painless script:
String datetime = '2022-01-13T10:42:10.982Z';

ZonedDateTime zdt = ZonedDateTime.parse(datetime);

long milliSinceEpoch = zdt.toInstant().toEpochMilli();

return milliSinceEpoch

I tried following scripts

1. "script": {
   "source": """ String datetime = ctx['TimeStamp'];
   ZonedDateTime zdt = ZonedDateTime.parse(ctx['TimeStamp']);
   long milliSinceEpoch = zdt.toInstant().toEpochMilli();
   ctx['test'] = milliSinceEpoch;
   """
2. "script": {
   "source": """ String datetime = ctx['TimeStamp'];
   ZonedDateTime zdt = ZonedDateTime.parse(ctx['TimeStamp']);
   long milliSinceEpoch = zdt.toEpochMilli();
   ctx['test'] = milliSinceEpoch;
   """

but I get same error..

[Rakesh_Rayabarapu (Rakesh R)]

I also tried with this script which leads different error. Please refer screenshot.

[Tomo_M (Tomohiro Mitani)]

it looks there is no 'datetime' in ctx. ctx[datetime] should be datetime, isn't it?

[Rakesh_Rayabarapu (Rakesh R)]

I tried even with TimeStamp that exist in ctx. same error.
Without ctx on timestamp

with ctx on timestamp

[Tomo_M (Tomohiro Mitani)]

From next time please share your code using </> button to be preformatted.

Maybe you have to add Zone information to timestamp.
Using DateTime in Painless is quite important to deal with datetime in painless.

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {"script": {
        "source": """DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS");
        LocalDateTime ldt = LocalDateTime.parse(ctx['TimeStamp'], dtf);
        ZonedDateTime zdt = ldt.atZone(ZoneId.of('Z'));
        ctx['test']=zdt.toInstant().toEpochMilli();"""
      }}
    ]
  },
  "docs":[
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "TimeStamp": "2022-01-13T10:42:10.982"
      }
    }]
}

if you can change timestamp to including timezon like "TimeStamp": "2022-01-13T10:42:10.982Z", it is more simple.

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {"script": {
        "source": """
        ZonedDateTime zdt = ZonedDateTime.parse(ctx['TimeStamp']);
        ctx['test']=zdt.toInstant().toEpochMilli();"""
      }}
    ]
  },
  "docs":[
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "TimeStamp": "2022-01-13T10:42:10.982Z"
      }
    }]
}

[Rakesh_Rayabarapu (Rakesh R)]

@Tomo_M

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {"script": {
        "source": """DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS");
        LocalDateTime ldt = LocalDateTime.parse(ctx['TimeStamp'], dtf);
        ZonedDateTime zdt = ldt.atZone(ZoneId.of('Z'));
        ctx['test']=zdt.toInstant().toEpochMilli();"""
      }}
    ]
  },
  "docs":[
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "TimeStamp": "2022-01-13T10:42:10.982"
      }
    }]
}

It worked like magic..
Thank you so much Tomo for your time and response.

[Rakesh_Rayabarapu (Rakesh R)]

I have extended the solution when "TimeStamp": "2022-01-25T07:30:11.463+0000"

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "description": "Get EpochMilli from SampleStartTime",
          "lang": "painless",
          "source": """
            String[] TempSplit = ctx['TimeStamp'].splitOnToken('+');
            String Temp= TempSplit[0].trim();
            DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS");
            LocalDateTime ldt = LocalDateTime.parse(Temp, dtf);
            ZonedDateTime zdt = ldt.atZone(ZoneId.of('Z'));
            ctx['SampleStartTime_millis']=zdt.toInstant().toEpochMilli();
            """,
            "ignore_failure": true
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "TimeStamp": "2022-01-25T07:30:11.463+0000"
      }
    }
  ]
}