# Extracting time from @timestamp field using Runtime

**URL:** <https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468>\
**Category:** Kibana\
**Tags:** runtime\
**Created:** [December 2, 2023, 7:53am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468 "2023-12-02T07:53:37Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 2, 2023, 7:53am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/1 "2023-12-02T07:53:37Z")

</div>

I understand Scripted Fields have been deprecated since 7.13.

The replacement is now Runtime.

I have a `@timestamp` field  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/a/7a2f78b03e5ad7ae9e820c908c13a5ba9acaf643.png)

How can I extract out the time component into a new field `@timeofday` - such that when I filter by @timeofday, I can filter by time range.

I was reading

[Lucene Expressions Language | Elasticsearch Guide [5.0] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/modules-scripting-expression.html#_date_field_api)

and

> **[Map a runtime field | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/runtime-mapping-fields.html)**

But still feel abit lost on how complicated an approach I need to do. Its actually a simple mod because I'm relying on data in existing columns to derive new ones.

**Index Management**

Since this new field is an in-line addition. @timeofday is non-existent in my Index Template.

Should I ingest more months of data (1 index per month), how can ensure this new `@timeofday` field will be calculated and included in the new data so that when I Discover, everything is seamless / consistent.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 2, 2023, 3:35pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/2 "2023-12-02T15:35:03Z")

</div>

What version are you on?

Also what do you want for Time of Day?

`HH:mm:ss`?

If you want to do a range, it will need to be something more like Number ... so I am not sure exactly what you want... I will take a look.

> **[Shared API for package java.time | Painless Scripting Language \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-time.html)**

> - int [getHour](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/ZonedDateTime.html#getHour())()

In the Data View it is pretty easy

 ![Screenshot 2023-12-02 at 7.48.17 AM](https://us1.discourse-cdn.com/elastic/original/3X/7/6/760f331640ea4c1448c7c2f98f64d160ab060cef.png)

You can test it

Then if you really want to use it in queries etc... then you need to actually add it to the mapping / template... from the docs you link above using a runtime field not the lucene expression. (BTW you linked to a really old document)

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 2, 2023, 4:06pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/3 "2023-12-02T16:06:34Z")

</div>

Sorry.

I'm now on 8.11

Yes `HH:mm:ss` is what im looking for

> Then if you really want to use it in queries etc... then you need to actually add it to the mapping / template

Sounds like I will need to do a re-indexing of my 6 months of data just to update my mapping / template with this new @timestamp field?

If I use the console to do the Update, i understand the existing data doesn't get updated with this new field?

Since the last time we discussed during the setup phase, I've continued to keep my data only at 6 months worth.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 2, 2023, 4:55pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/4 "2023-12-02T16:55:18Z")

</div>

> [@Ethan777100](#):
>
> Yes `HH:mm:ss` is what im looking for

So you need to look at examples here...

> **[Using Datetime in Painless | Painless Scripting Language \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-datetime.html)**

And the API Here...

> - [java.time](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-time.html)
> - [java.time.chrono](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-time-chrono.html)
> - [java.time.format](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-time-format.html)
> - [java.time.temporal](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-time-temporal.html)
> - [java.time.zone](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-java-time-zone.html)

OK we can do that as a keyword... filtering and sorting you will need to see if that works... You will need to figure out the timezone stuff if you want to...

```auto
ZonedDateTime zdt = doc['@timestamp'].value;
String datetime = zdt.format(DateTimeFormatter.ISO_LOCAL_TIME);
emit(datetime);

```

 ![Screenshot 2023-12-02 at 8.54.19 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2fbf4ab83219db736ff5d3a1b87670c67105b191.png)

 ![Screenshot 2023-12-02 at 8.54.32 AM](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6c4f6a85d30160411bb6cb937aed974acba0907.png)

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 2, 2023, 5:27pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/5 "2023-12-02T17:27:00Z")

</div>

I just tried it out.

`Keyword` is not my desired type.

Need it to be in Date format so that you can filter in between 2 timings that will just be `HH:mm:ss`

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f8f75c7f25e96c1c1bfdd794b0983ec30cc71ca3.png)

Yeah the timezone looks tricky to account for

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 2, 2023, 5:40pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/6 "2023-12-02T17:40:56Z")

</div>

> [@Ethan777100](#):
>
> Need it to be in Date format so that you can filter in between 2 timings.

🙂 Just and Hour / Minutes is not a Date format....

You are not going to be able to use that ....

You can create a custom filter... using a [range query filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html)

You can read about that

```auto
{
  "bool": {
    "must": [
      {
        "range": {
          "time_of_day": {
            "gte": "15:51:45.468"
          }
        }
      }
    ]
  }
}

```

 ![Screenshot 2023-12-02 at 9.37.49 AM](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ea113aa455d93a714a9751a72ba9730fd750c408.png)

 ![Screenshot 2023-12-02 at 9.38.52 AM](https://us1.discourse-cdn.com/elastic/original/3X/e/d/edcef0b92039fa19cd3a5e7f17eb3bb6dd77d7d0.png)

 ![Screenshot 2023-12-02 at 9.41.23 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/0/208f9e455989b73df1ed438821b6bba2ec25c68e.png)

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 2, 2023, 5:42pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/7 "2023-12-02T17:42:41Z")

</div>

I will try explore this.

Basically with my 6 months of logs, which contain entries 24/7

I want to do such that I filter away entries lasting from 0100-0430hrs in early morning, for every day (ie. 1 Jan - 30 June) (and likewise scaling this to more days when I pipe in additional data.

I wanted my Data View to omit 0100-0430hrs for every day.

EDIT: I tried using @timestamp to Query DSL but it didn't work. Wasn't as straight fwd to only read the time component.

Do I really need the new column to do this filtering?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 2, 2023, 5:55pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/8 "2023-12-02T17:55:20Z")

</div>

> [@Ethan777100](#):
>
> Do I really need the new column to do this filtering?

Welll Depends on what you actually want to do... as you are only providing partial information 🙂

If you want to filter ... yes you will need to create another field

If you want to aggregate you could use a Date Histogram / Visualization then no you do not

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 2, 2023, 6:05pm UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/9 "2023-12-02T18:05:57Z")

</div>

Logically I want to filter **for** entries NOT within 00:30:00 and 05:00:00

I used must\_not and it worked

![image](https://us1.discourse-cdn.com/elastic/original/3X/2/5/2508737e3da3037322a7f4fd175d6714a0296bda.png)

DSL

```auto
{
  "bool": {
    "must_not": [
      {
        "range": {
          "@time_of_day": {
            "gte": "00:30:00.00",
            "lte": "05:00:00.00"
          }
        }
      }
    ]
  }
}

```

Runtime field

```auto
ZonedDateTime zdt = doc['@timestamp'].value;
ZonedDateTime Updatedzdt = zdt.plusHours(8);
String datetime = Updatedzdt.format(DateTimeFormatter.ISO_LOCAL_TIME);

emit(datetime);

```

I can appreciate the result and extent of convenience. So the new field is only present in that specific dataview.

Ok, I'm now thinking further if I could try integrating this into Index Template and make `@time_of_day` a permanent "scripted" column that will exist inside the Index Template.

It will extract the `hh:mm:ss.xxx` component from `@timestamp` and translate to the new column `@time_of_day`

But yet in the raw csv files, the `@time_of_day` column will not exist.

Possible can this be done?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 4, 2023, 4:52am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/11 "2023-12-04T04:52:14Z")

</div>

> [@Ethan777100](#):
>
> It will extract the `hh:mm:ss.xxx` component from `@timestamp` and translate to the new column `@time_of_day`
> 
> But yet in the raw csv files, the `@time_of_day` column will not exist.

Add the mapping to the template and a `script` process to the ingest pipeline... and you do not need the `@` for every timestamp / date that is just and nameing convtion for the special field `@timestamp`

take a look at this...

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """ZonedDateTime zdt = ZonedDateTime.parse(ctx['@timestamp'], DateTimeFormatter.ISO_ZONED_DATE_TIME);
          String time_of_day = zdt.format(DateTimeFormatter.ISO_LOCAL_TIME);
          ctx['time_of_day'] = time_of_day;
          """
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "@timestamp": "2023-12-04T03:55:39.219Z"
      }
    }
  ]
}

# result

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "time_of_day": "03:55:39.219",
          "@timestamp": "2023-12-04T03:55:39.219Z"
        },
        "_ingest": {
          "timestamp": "2023-12-04T04:50:52.377145525Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 4, 2023, 4:59am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/12 "2023-12-04T04:59:12Z")

</div>

I modified my field script slightly to address the timezone offset of 8 hours. This checked well for me.

```auto
ZonedDateTime zdt = doc['@timestamp'].value;
ZonedDateTime Updatedzdt = zdt.plusHours(8);
String datetime = Updatedzdt.format(DateTimeFormatter.ISO_LOCAL_TIME);

emit(datetime);

```

Let me try your code later for mapping and ingest.

On this note, I understand that even if I update the mapping, existing data will not be re-indexed by the new change?

Because say if I remove the `@time_of_day` from Discover, the current dataset from Jan-June 2023 will not have this column and will not pick up the new `time_of_day` that's now present in Index Template?

I come from a "housekeeping / consolidation" pov, if I already had a permanent code inside Index Template, then I technically won't need the "makeshift" field I created on Discover which is at a more front end portion of things.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 4, 2023, 6:43am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/13 "2023-12-04T06:43:35Z")

</div>

You can add a runtime field to the mapping then you do not need to reload the data.

Or you can use an ingest pipeline and reload the data.

Pretty much your two choices

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [December 4, 2023, 6:45am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/14 "2023-12-04T06:45:45Z")

</div>

Is This Console entry considered the Runtime method?

Even tho its

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {

```

I still see

```auto
"script": {
          "lang": "painless",

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 4, 2023, 6:47am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/15 "2023-12-04T06:47:56Z")

</div>

No as the API indicates that's an ingest pipeline so that happens at ingest.

Feel free to read the docs...

There is a section on runtime fields

And there's another section on ingest pipelines.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2024, 6:48am UTC](https://discuss.elastic.co/t/extracting-time-from-timestamp-field-using-runtime/348468/16 "2024-01-01T06:48:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.