Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier

Need a little help with this one. Tried doing as the deprecation log states, but logstash crashes on restart.

The full deprecation log message is:

'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.

Working logstash config in 6.8:

                date {
                locale => "en"
                match => ["logTimestamphttp", "dd/MMM/yyyy:HH:mm:ss Z"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
                date {
                locale => "en"
                match => ["logTimestampString", "YYYY-MM-dd;HH:mm:ss.SSS"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }

Config changes I've made listed here with the result:
-This results in no logs reaching my existing indexes after restarting logstash.

                date {
                locale => "en"
                match => ["logTimestamphttp", "dd/MMM/8yyyy:HH:mm:ss Z"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
                date {
                locale => "en"
                match => ["logTimestampString", "8yyyy-MM-dd;HH:mm:ss.SSS"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }

-This results in logstash not restarting.

                date {
                locale => "en"
                match => ["logTimestamphttp", "dd/MMM/8uuuu:HH:mm:ss Z"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
                date {
                locale => "en"
                match => ["logTimestampString", "8uuuu-MM-dd;HH:mm:ss.SSS"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }

-So, I start thinking, "oh! prefix the date! I'll move the 8 to the beginning of the whole date!" Again, this results in no logs reaching my existing indexes after restarting logstash.

                date {
                locale => "en"
                match => ["logTimestamphttp", "8dd/MMM/yyyy:HH:mm:ss Z"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
                date {
                locale => "en"
                match => ["logTimestampString", "8yyyy-MM-dd;HH:mm:ss.SSS"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }

-"Okay, maybe I need to use the 'u' like the message says in the deprecation log". Still logstash crashes on restart.

                date {
                locale => "en"
                match => ["logTimestamphttp", "8dd/MMM/uuuu:HH:mm:ss Z"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
                date {
                locale => "en"
                match => ["logTimestampString", "8uuuu-MM-dd;HH:mm:ss.SSS"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
[2019-07-17T14:23:19,603][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalArgumentException", :message=>"Illegal pattern component: uuuu", :backtrace=>["org.joda.time.format.DateTimeFormat.parsePatternTo(org/joda/time/format/DateTimeFormat.java:566)", "org.joda.time.format.DateTimeFormat.createFormatterForPattern(org/joda/time/format/DateTimeFormat.java:687)", ...

What do I need to change in order for this move to 7.x handle my dates properly?

Am I changing the format in the wrong place? The "Breaking Changes" page doesn't really offer much more insight into this change and where those changes need to happen.

With the release of Elasticsearch 6.7 a backwards compatibility layer was introduced, that checked if you are using a Joda-Time based formatter, that is supported differently in java time. A log message was emitted, and you could create the proper java time based formatter prefixed with an 8 .
With Elasticsearch 7.0 all formatters are now java based, which means you will get exceptions when using deprecated formatters without checking the deprecation log in 6.7. In the worst case you may even end up with different dates.
An example deprecation message looks like this, that is returned, when you try to use a date formatter that includes a lower case Y
Use of 'Y' (year-of-era) will change to 'y' in the next major version of Elasticsearch. Prefix your date format with '8' to use the new specifier.
So, instead of using YYYY.MM.dd you should use 8yyyy.MM.dd .
You can find more information about available formatting strings in the DateTimeFormatter javadocs.

Also tried with:

            date {
                locale => "en"
                match => ["logTimestamphttp", "dd/MMM/yyyy:HH:mm:ss Z"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }
                date {
                locale => "en"
                match => ["logTimestampString", "yyyy-MM-dd;HH:mm:ss.SSS"]
                timezone => "America/New_York"
                target => "logTimestamp"
            }

which also results in the same deprecation log.

Bump. Still seeking direction with this issue.

any ideas? Still trying to address this to get upgraded to 7.x

So in 6.8 you used yyyy-MM-dd;HH:mm:ss.SSS fine, but got the deprecation warning? Have you tried the same format on 7.0? y should now be used for year-of-era, but I think as long as you stay in the current era it shouldn't really make a difference?

Alternatively, use uuuu-MM-dd;HH:mm:ss.SSS with 7.0?

I haven't tried this yet, but I wonder whether this is really such a big deal?

Thanks for the reply. I am currently on 6.8 and trying to resolve the deprecation message before upgrading. According to the documentation, it is a big deal:

which means you will get exceptions when using deprecated formatters without checking the deprecation log in 6.7. In the worst case you may even end up with different dates.

I can't afford exceptions or incorrect dates in our production environment. I am hoping someone can offer a direct answer to resolve the deprecation message prior to moving to 7.x and having to fight with exceptions or incorrect dates.

I just tried to recreate this issue with a simple config:

input {
  tcp {
    port => 5555                                                                                                                                                                                    
  }
}

filter {
  date {
    match => ["message","dd/MMM/yyyy:HH:mm:ss Z"]
  }
}

output {
  stdout {}
}

This doesn't show deprecation warning for me, using Logstash 6.8.1.
My next attempt would have been to check this config with 7.2 and see what changes?

Thank you for the update. Are you checking the elasticsearch deprecation log for the error?

To test, I've removed all configurations from the logstash conf.d directory and loaded only the one you pasted. I restart logstash to load the new configuration. Once logstash is started, I restart elasticsearch. I'm still seeing a warning in the deprecation log.

[2019-07-25T09:35:41,601][WARN ][o.e.d.c.j.Joda           ] [hostname.domain] 'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.

To me, this means 1 of 3 things:

  • this warning isn't checking if the date format in the config is actually correct. However, if you tested this in logstash and checked the deprecation log, that disproves this idea.
  • I have a configuration somewhere else that is causing this issue that isn't the logstash configuration.
  • there is a bug in the software.

I've double checked every configuration file, including the yml files to make sure there's nothing else being pulled in. I turned on debug logging for deprecation in the elasticsearch logging properties, but that doesn't add anything to the deprecation log.

I also completely stopped logstash and still see the warning in the elasticsearch deprecation log.

[2019-07-25T09:48:46,380][WARN ][o.e.d.c.j.Joda           ] [hostname.domain] 'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.
systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2019-07-25 09:44:01 EDT; 5min ago

I'm starting to believe that this warning isn't actually checking anything before presenting itself. Can someone confirm that they do/don't see this warning in their elasticsearch deprecation log?

Wait a second - this warning comes up on Elasticsearch side or Logstash side?

This is an elasticsearch deprecation log warning.

Then you cannot fix this in Logstash.
I think the error would come up with ingest pipelines, which are set up in elasticsearch. Depending on what other shippers you use, this might occur with an older Filebeat version: https://github.com/elastic/beats/issues/10433

Thank you for the reply. I'm definitely using an older version of filebeats. I'll update the version of filebeats I'm using to test and see if that resolves the warning. I'll check back in with any updates.

I was able to update the version of filebeat to 7.2.0. After doing so, I changed the logstash configuration back to the basic one provided, then restarted logstash. Then, restarted elasticsearch. The same warning still appears in the deprecation log:

[2019-08-01T11:45:41,575][WARN ][o.e.d.c.j.Joda           ] [hostname.domain] 'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.

I checked my ingest pipeline like you mentioned. I can find nothing related to the warning message elasticsearch is printing in the deprecation log (above).

GET _ingest/pipeline
{
  "xpack_monitoring_2" : {
    "description" : "This pipeline upgrades documents from the older version of the Monitoring API to the newer version (6) by fixing breaking changes in those older documents before they are indexed from the older version (2).",
    "version" : 6070299,
    "processors" : [
      {
        "script" : {
          "source" : "boolean legacyIndex = ctx._index == '.monitoring-data-2';if (legacyIndex || ctx._index.startsWith('.monitoring-es-2')) {if (ctx._type == 'cluster_info') {ctx._type = 'cluster_stats';ctx._id = null;} else if (legacyIndex || ctx._type == 'cluster_stats' || ctx._type == 'node') {String index = ctx._index;Object clusterUuid = ctx.cluster_uuid;Object timestamp = ctx.timestamp;ctx.clear();ctx._id = 'xpack_monitoring_2_drop_bucket';ctx._index = index;ctx._type = 'legacy_data';ctx.timestamp = timestamp;ctx.cluster_uuid = clusterUuid;}if (legacyIndex) {ctx._index = '<.monitoring-es-6-{now}>';}}"
        }
      },
      {
        "rename" : {
          "field" : "_type",
          "target_field" : "type"
        }
      },
      {
        "set" : {
          "field" : "_type",
          "value" : "doc"
        }
      },
      {
        "gsub" : {
          "field" : "_index",
          "pattern" : "(.monitoring-\\w+-)2(-.+)",
          "replacement" : "$16$2"
        }
      }
    ]
  },
  "xpack_monitoring_6" : {
    "description" : "This is a placeholder pipeline for Monitoring API version 6 so that future versions may fix breaking changes.",
    "version" : 6070299,
    "processors" : [ ]
  }
}

NOTE: In my case, using the configuration from @BennyInc would take filebeat completely out of the equation because I'm using certificates (and a different port) and filebeat won't be able to communicate with logstash without the certificate configuration (or correct port). So, this appears to boil down to something between logstash and elasticsearch. Or, something with elasticsearch by itself.

To test this further, I stopped logstash completely, then restart elasticsearch. I still receive the warning message. So, now it seems limited to elasticsearch only. Any other ideas of what to check, or why this message might be appearing in the deprecation log for elasticsearch still?

Any suggestions are appreciated. If anyone can confirm whether their elasticsearch deprecation log contains the same warning on restart, that would be helpful too.

Installed fresh elasticsearch 6.8.2 on a new server. After starting, I still see the same warning. Appears this warning won't check the current configuration format and just spits out this message regardless of the current formatting being used. Upgraded this installation to 7.3. That clears the deprecation warning.

However, that doesn't lessen my concern about this warning message and the consequences of improper formatting for an unknown date format. Most concerning, is that it's not even clear where that formatting needs to be changed.

I'm going to move forward with updating my test environment to see if I can make sense of this warning message. I will update with my findings once I am able to do so.

Did you check the Upgrade Assistant in Kibana? It might provide more information.

I did. That was the initial step before moving forward to checking deprecation logs. Unfortunately, it didn't list anything valuable.

I will add that I've successfully upgraded to 7.3 at this time. I am still inspecting the ingested logs vs what is expected to test that there aren't any dates broken in the cluster.

hey,

quite the thread already. let me try to untangle a little bit, as there is some confusion between logstash and elasticsearch here and I hope that clears some of the fog. I'd super appreciate if you help us with improving the docs in this case, as you are probably not the only one stumbling over this. Sorry for the experience so far. So let's get rolling:

To add some backstory: In Elasticsearch 7.0 we replaced joda time with the JDK built-in java time classes in most of our codebase. The reasoning behind this was to get rid of a library over time that is on maintenance mode because of its successor being right in the JDK. Also, switching to java.time allowed us to introduce nanosecond granularity, which was impossible with joda time, as it internally stored the date as a long representing the milliseconds since the epoch. In 6.8 we had to introduce a compatibility layer to be able to handle joda time dates as well as java time dates, so that things like scripting or upgrading would be smooth.

All of this happened only Elasticsearch. Logstash is still using joda time, as mentioned in the date filter. All the logstash configurations need to remain in joda syntax.

However, my assumption here is, that some index template is still configured with a joda based timezone and it might be that logstash is trying to store this template.

Can you maybe take a look at your templates and search for that date string and see if it occurs somewhere? Retrieving all the templates and grepping for yyyy should be a good start.

Hope that clears a few things up.

1 Like

Thank you for jumping in here.

Based on the explanation, it sounds like the warning is legitimate and should be addressed before upgrading. What's concerning is that I got the same message on a fresh install of elasticsearch 6.8 as well, and it was only cleared after upgrading to 7.x. It's not clear if this warning is actually checking that anything is wrong before presenting itself. Any clarity on this, or some logic that would not present the message if there is no conflict, would be helpful.

I should have mentioned earlier that one of the first things I did was check my elasticsearch template to ensure there was no date format being set there.

By process of elimination I've ended up determining exactly what you've explained. That this is isolated to elasticsearch, and does not involve anything else (filebeat/logstash).

I'm becoming more convinced that my cluster did not have a conflict to begin with based on the troubleshooting steps I've outlined here. Is there anything else that we can check that might give us more understanding? Can you explain if this warning message is actually checking any files/configs? And, if so, which ones? I could double check all of those if we have a list. And, it would be a good place to start for anyone else running into this.