Logstash dateparse error

I have a logstash pipleline working from years. worked since 7.1 till 8.5.3 (no problem on any upgrade)

two month ago when I upgraded to 8.5.3 it was still working fine.

suddenly it stop working due to date parsing failed. It stoped working since 9/1/2023

after a lot of debugging this is what I discover:

This works on 7.x logstash, but not on 8.5.3 (Server is sitting in UK)
Same setup works in US system on any version.

mutate { add_field => { "sub" => "03-SEP-23 15:50:15" } }
   date {
       match => ["sub", "dd-MMM-yy HH:mm:ss", "ISO8601"]
      target => "sub"

but as soon as I add locale it works in UK

 mutate { add_field => { "sub" => "03-SEP-23 15:50:15" } }
   date {
        locale => "en"
      match => ["sub", "dd-MMM-yy HH:mm:ss", "ISO8601"]
      target => "sub"

Anyone knows what I need to setup to make it work without locale setting.

For me logstash just stops working out of blue is something I can't understand

system has proper local setup

#  cat /etc/locale.conf

It does sound like a locale problem. If I run this configuration in an en_US locale

input { generator { count => 1 lines => [ '' ] } }
output { stdout { codec => rubydebug { metadata => false } } }
filter {
    mutate { add_field => { "date1" => "03-Sep-23 15:50:15" } }

    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateA" locale => "en_GB.UTF-8" }
    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateB" locale => "en_GB" }
    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateC" locale => "en_UK" }
    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateD" locale => "en" }

then I get [dateA], [dateC], [dateD], plus a _dateparsefailure instead of [dateB].

I have no clue how $LANG gets translated into the default locale by the time Ruby calls Java. Using locale => "$LANG" might be an option.

1 Like

I actually discover by looking at one of your very old post last night while debugging this issue.

it still works if I try exact same thing on 7.x logstash.

and LANG=en_US.UTF-8 works as well even with 8.x logstash

any explanation on why it stop working

but still works with older logstash?

and yes by the way when I use local => "${LANG}" it works

Whoever might face this problem in feature. I am posting this for them

basically I was reading data with dd-MMM-yy and now jave is using BCP47 locale codec.

BCP47 locale is mostly necessary to be set for parsing month names (pattern with MMM) and
weekday names (pattern with EEE).

when I change my input retrieval date from Relational database to DD-MM-YY then I don't need this local setting in my logstash.

@Badger thanks for testing this out which give me right direction to do more research.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.