Logstash - %{+YYYY} returns a year two years behind around year boundary – how is this possible?

Elastic Stack Logstash
1

I have a Logstash pipeline where I build a timestamp string manually and then parse it into event.timestamp because we do not have a year value in the log message, there are just day and month in the file name.

Environment / Timezone setup

  • Logstash runtime TZ = UTC
  • Local business timezone UTC+8
  • I do not set timezone in the date filter, timezone conversion is intentionally handled only at the presentation layer (Kibana)
  • @timestamp is NOT modified anywhere in the pipeline and looks correct in Kibana.
  • The system ingests real-time data.

Pipeline code:

filter {

  grok {

    match => {

      "message" => "^(?<seq_hour>\d{2})(?<seq_minute>\d{2})(?<seq_second>\d{2})"

    }

  }

  mutate {

    add_field => {

      "log_timestamp_str" =>

        "%{+YYYY}-%{file_month}-%{file_day} %{seq_hour}:%{seq_minute}:%{seq_second}"

    }

  }

  date {

    match  => ["log_timestamp_str", "yyyy-MM-dd HH:mm:ss"]

    target => "[event][timestamp]"

  }

}

Observed behavior:

When the local time is 7 AM on 2026-01-01, file_month = 01, file_day = 01, time in log = 07:00:00.

I get @timestamp:

2025-12-31T23:00:00.456Z

Resulting event.timestamp becomes

2024-12-31T23:00:00.000Z

@timestamp itself appears correct in Kibana devtool and is not being modified.

Now, I understand this can be caused by Timezone issue because, the issue disappears when the local time comes to 2026-01-01 09:00:00 (out of 8 hour boundary). Just not sure why the year becomes 2024 in this case.

My Questions:

  1. Is %{+YYYY} in Logstash always derived from @timestamp at the time the mutate filter runs?
  2. This is confusing because, as far as I understand, 2025-12-31 (calendar year) should only map to ISO week-year 2025 or 2026. Why can an ended ISO week-year of 2025 be matched to 2024 (Please help me correct if I am wrong about this)?

Any clarification from Elastic engineers or references to official documentation would be much appreciated.

2

What is your input? Filebeat/agent perhaps?

3

My input is Agent with Custom Log integration (filestream).

4

Check EA, actually your logs, @timestamp might be from log on EA/FB side. You can check in logs or output.console on EA side.

If you want LS time, use ruby:

    ruby {
        code => ' time = Time.now
        event.set("ls-time", time)'
    }

Output:

{
       "ls-time" => 2026-01-05T11:14:43.643Z,
    "@timestamp" => 2026-01-05T11:14:43.615757200Z,
}
  1. Is %{+YYYY} in Logstash always derived from @timestamp at the time the mutate filter runs?

Yes, from @timestamp.

5

Unless there is anything else in the pipeline that you didn't share, the value of @timestamp is unrelated to the value of log_timestamp_str and event.timestamp.

Can you share the raw value of log_timestamp_str ? Where @timestamp is coming from? Do you have other date filters?

Using date sprintf like %{+YYYY} in Lostash will always use the value of @timestamp.

You need to share more context on how @timestamp is being generated, but since your events do not have an year, an event on 01/01 that would be 2026-01-01 may be interpreted as 2025-01-01, which depending on the times can be 2024-12-31 in UTC.

Can you share the raw logs of an event where this happened and also provide context on how @timestamp is being generated?

1 Like
6

Yes, always. But logstash just calls Joda to do the parsing. I don’t know if it is documented, but that looks like you could also try a Java (%{{yyyy}}) format to get a different parser.

If you were using a date filter to set the timestamp (and I hear that you say that are not) then there would be other possible issues around heuristics used to guess years from timestamps that do not include the year.

7

@timestamp field is generated automatically (afaik, from Agent machine’s clock value at UTC). It is not from the log message, and do not define it in the code or customize it anywhere.

Can you share the raw logs of an event where this happened and also provide context on how @timestamp is being generated?

The issue occurred within the local time range 2026-01-01 00:00:00 to 2026-01-01 08:00:00 (UTC+8):

During this window, anomalies began: @timestamp values remained correct in UTC (2025-12-31 16:00:00), but event.timestamp’s year started behaving oddly (2024-12-31 16:00:00).

8

Yes, i do not specify @timestamp anywhere, afaik, it should be generated automatically from Agent side. Agent servers and Logstash servers are in the same time zone. I just do not know why @timestamp are all correct but event.timestamp are not while the year of event.timestamp was built from @timestamp’s year.

9

This does not matter much as they will use UTC for dates, the main issue would be if the dates does not have the timezone information.

I don't think there is any issue here, the event.timestamp is created combining information from the @timestamp field and fields generated by parsing the message, but those dates seems to be in different timezones, the @timestamp will be in UTC and the month and day field are in your local timezone, this will certainly adds confusion.

Do you really have documents where the field event.timestamp has the value 2024-12-31T23:00:00.000Z?

The value you see in Discover in the table view mode is not on UTC, to get the UTC value you need to expand the document and look at it and share.

Please share an example from Kibana where you have this date differece, share both the visualization in Discover, and then expand the document and share the json data for it, this will show how the date is stored.

10

No, I did not view it in Discover because I knew the time there depends on Kibana's timezone settings. I viewed the raw documents in the DevTools console by querying elasticsearch API. Unfortunately, one of my team members removed all the data during that day, so I am unable share it now. :frowning:

The observed values of documents were exactly what I mentioned in my question (Observed behavior).

11

Yeah, but without seeing how the raw events it is not possible to confirm what happened, you also didn't share your full pipeline.

Also, based on the part of the pipeline you shared, the value of the event.timestamp does not make any sense.

You mentioned that when you have this:

When the local time is 7 AM on 2026-01-01, file_month = 01 , file_day = 01 , time in log = 07:00:00 .

The event.timestamp became 2024-12-31T23:00:00.000Z, but with just what you shared this is not possible.

Your log_timestamp_str would be a string like this: 2025-01-01 07:00:00, and your date filter does not have any timezone information, so it would interpret this date as UTC and would populate the event.timestamp with 2025-01-01T07:00:00Z, this would be the indexed value, in Kibana this would show up as 2024-12-31 23:00:00, but would not be indexed as 2024-12-31T23:00:00.000Z in the way you shared.

For this to be indexed as 2024-12-31T23:00:00.000Z you would need another date filter in Logstash or Elasticsearch Ingest Pipeline to apply the timezone.

This was explained, your logic to create log_timestamp_str only considers the year from @timestamp and the day, month and time comes from a log on a different timezone, this results in an edge case that will happen every time the year changes.

Can you share some recent events, both the raw data before indexing and how they are indexed in Elasticsearch? Seeing how current dates are being indexed may make possible to backtrack what happened and explain, but the issue is related to the logic to create the event.timestamp field.

But without the full pipeline and sample events it is not possible to replicate.

1 Like
12

@josh_tran I am not clear what you are now expecting from the thread, if anything? Can you confirm.

I don't see any evidence you have hit/found some bug, so likely user error or bad data or some misunderstanding, or some combination thereof. If you need help to find that you really need supply more information, including the full pipeline and sample events, including some of those that were problematic.

1 Like