Logstash dateparse error

Elastic Stack Logstash
1

I have a logstash pipleline working from years. worked since 7.1 till 8.5.3 (no problem on any upgrade)

two month ago when I upgraded to 8.5.3 it was still working fine.

suddenly it stop working due to date parsing failed. It stoped working since 9/1/2023

after a lot of debugging this is what I discover:

This works on 7.x logstash, but not on 8.5.3 (Server is sitting in UK)
Same setup works in US system on any version.

mutate { add_field => { "sub" => "03-SEP-23 15:50:15" } }
   date {
       match => ["sub", "dd-MMM-yy HH:mm:ss", "ISO8601"]
      target => "sub"
   }

but as soon as I add locale it works in UK

 mutate { add_field => { "sub" => "03-SEP-23 15:50:15" } }
   date {
        locale => "en"
      match => ["sub", "dd-MMM-yy HH:mm:ss", "ISO8601"]
      target => "sub"
   }

Anyone knows what I need to setup to make it work without locale setting.

For me logstash just stops working out of blue is something I can't understand

system has proper local setup

#  cat /etc/locale.conf
LANG="en_GB.UTF-8"
2

It does sound like a locale problem. If I run this configuration in an en_US locale

input { generator { count => 1 lines => [ '' ] } }
output { stdout { codec => rubydebug { metadata => false } } }
filter {
    mutate { add_field => { "date1" => "03-Sep-23 15:50:15" } }

    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateA" locale => "en_GB.UTF-8" }
    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateB" locale => "en_GB" }
    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateC" locale => "en_UK" }
    date { match => ["date1", "dd-MMM-yy HH:mm:ss"] target => "dateD" locale => "en" }
}

then I get [dateA], [dateC], [dateD], plus a _dateparsefailure instead of [dateB].

I have no clue how $LANG gets translated into the default locale by the time Ruby calls Java. Using locale => "$LANG" might be an option.

1 Like
3

@Badger
I actually discover by looking at one of your very old post last night while debugging this issue.

it still works if I try exact same thing on 7.x logstash.

and LANG=en_US.UTF-8 works as well even with 8.x logstash

4

any explanation on why it stop working

but still works with older logstash?

and yes by the way when I use local => "${LANG}" it works

5

Whoever might face this problem in feature. I am posting this for them

basically I was reading data with dd-MMM-yy and now jave is using BCP47 locale codec.

BCP47 locale is mostly necessary to be set for parsing month names (pattern with MMM) and
weekday names (pattern with EEE).

when I change my input retrieval date from Relational database to DD-MM-YY then I don't need this local setting in my logstash.

@Badger thanks for testing this out which give me right direction to do more research.