Mapping and Scripting

Elastic Stack Kibana
1

I have two fields named "kafkaTimestamp" and "sparkprocesstime" I want to script the difference between those two fields. The mapping of the two fields are as follows:

kafka_timestamp

"kafka_timestamp" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }

Spark process time

"spark_process_time" : {
            "type" : "date"
          }

Now I want know how I can convert the kafka time stamp into a date and then I want to look for the difference between those two by writing a scripted field.

2

Hi there, to do this you'll need to parse your Kafka timestamp into epoch time in milliseconds. I'm not sure of the format of your timestamp so you'll need to refer to the SimpleDateFormat docs for the correct pattern to use, but here's an example of how you'd like that:

def kafka = new SimpleDateFormat('YYYY-MM-DD:HH:mm:ss.SSS').parse(doc['kafka_timestamp'].value).getTime();
def spark_process_time = doc['spark_process_time'].value;
return spark_process_time - kafka_timestamp; // Difference between the two in milliseconds

You don't need to convert the spark process time because that will already be in epoch time in milliseconds (date field docs).

See this thread for more info and a link to the SimpleDateFormat docs: Converting a string date to a Date field using scripted fields in kibana

3

Hello @cjcenizal. Thank you for the reply. One last question.Is it possible in kibana version 6.5??
And if so I will try it outb and get back to you if any.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.