Return (emit) @timestamp date value in runtime field

nilei · July 29, 2022, 4:08pm

Dear all,

I am trying to output the value of @timestamp in a runtime field (configured in Kibana 7.13.3). My goal is then to get only the month as display value with a modified format 'MMM'. Unfortunately, I have been unsuccessful so far with all possible combinations that can be found on Google. I have not even managed to output the "normal" value of @timestamp or any other date field in a new runtime field.

emit(doc['@timestamp'].value);
                      ^---- HERE

In Discover, I then get this error message:

cannot convert MethodHandle(Dates)JodaCompatibleZonedDateTime to (Object)long

I also tried a gazillion of other things... with no success.

The same works great with a keyword field if I am using this:

emit(doc['user_name'].value);

How the heck does that work with dates ???

Thanks a lot for your help!!!

-- Cheers, Nils

stephenb · July 30, 2022, 4:22am

emit(doc['@timestamp'].value.toString())
2022-07-23T22:31:10.171Z

emit(doc['@timestamp'].value.month.getDisplayName(TextStyle.FULL, Locale.ROOT))
July

emit(doc['@timestamp'].value.month.getDisplayName(TextStyle.SHORT, Locale.ROOT));
Jul

ZonedDateTime zdt = ZonedDateTime.parse(doc['@timestamp'].value.toString());
emit(zdt.getMonthValue());
7 (integer)

ZonedDateTime zdt = ZonedDateTime.parse(doc['@timestamp'].value.toString());
emit(zdt.getMonthValue().toString());
7 (keyword)

emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))
Saturday

nilei · July 30, 2022, 2:21pm

Hey Stephen,

thanks a lot. This is a great resource of examples.

Could you help me further to return the value of @timestamp as a Date type in the runtime field? Using .toString(), which requires the runtime field to be a Keyword field, brings the problem that I cannot configure any custom date formatting, like MMM,

If I try to use emit(doc['@timestamp'].value.toString()) and runtime field type Date, the Discover app shows an error:

cannot convert MethodHandle(Object)String to (Object)long

-- Cheers, Nils

stephenb · July 30, 2022, 2:27pm

Apologies I am Confused what you want then

@timestamp is already a date

emit(doc['@timestamp'].value.toString()) is a string

Can you show me exactly what you want before and after and what data types you want before and after and how you plan to use the result and what type you want the result... I am lost / confused at this point.

nilei · July 30, 2022, 2:33pm

Hi Stephen,

thank you very much for your help!

I have data from about the last 10 years and would like an evaluation of which month has the most entries. As a date field I have the @timestamp field available. I need an evaluation of the data per month. Month should be for example "7" or "July". That would not matter. My plan was to create a runtime field, which gets the value of @timestamp and is displayed in the format MMM. So I still have the @timestamp field and additionally the month in a separate field.

-- Cheers, Nils

stephenb · July 30, 2022, 2:38pm

That is exactly what I showed you... confused... you have to name it as a new field..

You add a new field...this is in the Index Pattern if you which will work for visualizations.

BUT if you actually want to do searches / aggregation etc.. etc.. you need to add it to the mapping

PUT my-index-000001/
{
  "mappings": {
    "runtime": {
      "date_month_MMM": {
        "type": "keyword",
        "script": {
          "source": "emit(doc['@timestamp'].value.month.getDisplayName(TextStyle.SHORT, Locale.ROOT));"
        }
      }
    }

nilei · July 30, 2022, 2:43pm

Hi Stephen,

thanks for the clarification of the difference between mapping and index pattern. I am aware of this now. I also understand now how to return the month name as a string (Keyword runtime field), but I am still unsure how to return the value of @timestamp as a date (Date runtime field). Maybe I am just confused and the answer is too simple Sorry if that is the case...

-- Cheers, Nils

stephenb · July 30, 2022, 2:47pm

@timestamp is already always available as a "indexed / concrete field" as a date type... why do you need it as a runtime field? @timestamp will always be available.

What do you mean the the value of @timestamp as a date (Date runtime field)

Can you show me what you mean? Are you just trying to format the date in a certain format?

Are you trying to build visualizations or DSL queries?

nilei · July 30, 2022, 2:50pm

Hi Stephen,

I want to show and use the Month and the Year in separate additional fields. My idea is to keep the fields as Date fields and just apply a custom formatting as needed.

-- Cheers, Nils

stephenb · July 30, 2022, 2:53pm

Ok just create another field with the Year part... you can create a field if you like for every part of the date if you like.

nilei · July 30, 2022, 2:57pm

Hi Stephen,

let us just assume that I want to duplicate the @timestamp field into a runtime field. Let us call this runtime field my_date. How do I return the date value of the @timstamp field in the my_date field? The my_date field is a Date field type.

-- Cheers, Nils

stephenb · July 30, 2022, 3:34pm

Well then I would probably use an alias... not a runtime field...

And remember all dates are actually stored as epoch millis long... all you ever really see is the formatted date output...

I will look at how to duplicate a field later...

Think this will work, but put it as a mapping

ZonedDateTime zdt = ZonedDateTime.parse(doc['@timestamp'].value.toString());
emit(zdt.toEpochMilli());

I changed the name so I could see them side by side

I had to set the date formatters to look the same, That formats the output that is set in the Data View / Index Pattern

nilei · July 30, 2022, 4:04pm

Hi Stephen,

when you said ...

... that helped me a lot! I now have what I was looking for.

My solution is the following:

// Create a zoned datetime 
ZonedDateTime zdt = ZonedDateTime.parse(doc['@timestamp'].value.toString());

then:

// Return as long milli
emit(zdt.toInstant().toEpochMilli());

This discussion also helped me to find the correct syntax:

-- Cheers, Nils

nilei · July 30, 2022, 4:05pm

Hi Stephen,

great, this is exactly what I was looking for!

Thank you so much!

-- Cheers, Nils

stephenb · July 30, 2022, 4:13pm

Cool that is good too!!

Why you can not just turn around an emit the value is unclear to me... (bugging me actually)

nilei · July 31, 2022, 11:59am

That is what I was trying first, but without any success. So I thought that must be the wrong way.

-- Cheers, Nils

system · August 28, 2022, 12:00pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Runtime Field - How to create a field (scripted field) Kibana runtime	1	134	April 17, 2024
Extracting time from @timestamp field using Runtime Kibana runtime	14	1535	January 1, 2024
Converting String to Date in Kibana Kibana	3	1003	August 8, 2021
Aggregation using part of a timestamp field Kibana	5	538	July 21, 2022
[7.5.1] script field issue on @timestamp Kibana	7	3679	March 4, 2020

Return (emit) @timestamp date value in runtime field

Related topics