"_dateparsefailure" When trying to overwrite @timestamp field

Hi,
I am trying to solve this issue but don't seem to be having much luck. My log is as follows:

01-JAN-24 00:00:50|1.1.1.1|1|CN=test_user,OU=test Users,OU=test,OU=Business,DC=test,DC=corp,DC=abc,DC=ca|Z/dtEse7dwP2VEVRwULGNwhEur0=|App

filter
{
grok   {

pattern_definitions => {
            "CUSTOM_MONTH" => "(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)"
            "CUSTOM_TIMESTAMP" => "%{YEAR}-%{CUSTOM_MONTH}-%{MONTHDAY} %{TIME}"
        }
         match => { "message" => [ "%{CUSTOM_TIMESTAMP:time_stamp}\|%{IPV4:src_ip}\|%{WORD:event_id}\|%{GREEDYDATA:DN}\|%{NOTSPACE:session_id}\|%{WORD:application}" ] }


remove_field => ["message","host","business_unit","agent"]

         }
date {
        match => [ "time_stamp", "dd-MMM-yy HH:mm:ss" ]
        target => "@timestamp"
      }

}

I keep getting _dateparsefailure. Any suggetions? I feel like its something simple.

Can you share your logstash output?

It worked for me without any issue:

[2024-01-30T17:27:59,547][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
     "time_stamp" => "01-JAN-24 00:00:50",
             "DN" => "CN=test_user,OU=test Users,OU=test,OU=Business,DC=test,DC=corp,DC=abc,DC=ca",
       "@version" => "1",
     "@timestamp" => 2024-01-01T03:00:50.000Z,
    "application" => "App",
     "session_id" => "Z/dtEse7dwP2VEVRwULGNwhEur0=",
       "event_id" => "1",
         "src_ip" => "1.1.1.1"
}
[2024-01-30T17:27:59,724][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}

Ya its weird. If I try this on another system it works. I am running logstash on CentOS Stream 9 VM (Parallels) on a Macbook Pro with M1 Arch. If I switch to on older Macbook Running X86_64 it works with no issues.

[root@localhost bin]# ./logstash -f test.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2024-01-30 15:19:39.076 [main] runner - Starting Logstash {"logstash.version"=>"7.17.17", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-aarch64]"}
[INFO ] 2024-01-30 15:19:39.090 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2024-01-30 15:19:39.298 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2024-01-30 15:19:40.032 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2024-01-30 15:19:40.697 [Converge PipelineAction::Create<main>] Reflections - Reflections took 46 ms to scan 1 urls, producing 119 keys and 419 values 
[WARN ] 2024-01-30 15:19:41.104 [Converge PipelineAction::Create<main>] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2024-01-30 15:19:41.137 [Converge PipelineAction::Create<main>] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2024-01-30 15:19:41.145 [Converge PipelineAction::Create<main>] tcp - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2024-01-30 15:19:41.349 [[main]-pipeline-manager] grok - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2024-01-30 15:19:41.455 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/bin/test.conf"], :thread=>"#<Thread:0x4d1e5bc run>"}
[INFO ] 2024-01-30 15:19:41.837 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.38}
[INFO ] 2024-01-30 15:19:41.857 [[main]-pipeline-manager] tcp - Automatically switching from json to json_lines codec {:plugin=>"tcp"}
[WARN ] 2024-01-30 15:19:41.866 [[main]-pipeline-manager] jsonlines - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2024-01-30 15:19:41.910 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2024-01-30 15:19:41.932 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:8090", :ssl_enable=>false}
[INFO ] 2024-01-30 15:19:41.944 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2024-01-30 15:20:30.636 [nioEventLoopGroup-2-1] jsonlines - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
        "@timestamp" => 2024-01-30T09:30:10.799Z,
          "event_id" => "1",
          "@version" => "1",
                "DN" => "CN=test_user,OU=test Users,OU=test,OU=Business,DC=tes,DC=corp,DC=abc,DC=ca",
             "topic" => "application-test-ha",
        "time_stamp" => "01-JAN-24 00:00:50",
       "application" => "App",
              "site" => "test",
             "event" => {
        "timezone" => "-05:00"
    },
            "src_ip" => "1.1.1.1",
        "session_id" => "Z/dtEse7dwP2VEVRwULGNwhEur0=",
              "port" => 58956,
               "ecs" => {
        "version" => "1.0.0"
    },
              "tags" => [
        [0] "bep",
        [1] "_dateparsefailure"
    ],
    "config_version" => "1.1",
               "log" => {
        "offset" => 0,
          "file" => {
            "path" => "/apps/logs/soc/homeagents_ON_20210313.log"
        }
    },
             "input" => {
        "type" => "log"
    }
}


Yeah, not sure what is the issue, I tested on both Logstash 8.12 and 7.17.17 and got the same result.

I'm on Linux on a x86_64 system.

Maybe it is a weird bug related to Apple Silicon.

I would suggest that you report it on Github.

Thanks

Could it have a different locale set?

I think its working when I follow your other post and add locale => "$LANG"

date {
        locale => "$LANG"
        match => [ "time_stamp", "dd-MMM-yy HH:mm:ss" ]
        target => "@timestamp"
      }

        "time_stamp" => "30-JAN-24 00:00:50",
            "src_ip" => "1.1.1.1",
                "DN" => "CN=test_user,OU=test Users,OU=test,OU=Business,DC=tes,DC=corp,DC=abc,DC=ca",
        "@timestamp" => 2024-01-30T05:00:50.000Z,
             "event" => {
        "timezone" => "-05:00"
    },
2 Likes

I don't know is it useful, I have run LS 8.12.0 in the debug mode:

[2024-01-30T22:40:21,403][DEBUG][org.logstash.filters.DateFilter] Date filter with format=dd-MMM-yy HH:mm:ss, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser

The date has been parsed successfully, my regional settings are English.

The locale is mostly necessary to be set for parsing month names (pattern with MMM) and weekday names (pattern with EEE).

If not specified, the platform default will be used but for non-english platform default an english parser will also be used as a fallback mechanism.

Most likely _dateparsefailure is because of the moth naming. If you set locale => "es" or set to sl (Slovenia - correct is 01-januar-24) you will gen an error. Some localization do not accept JAN or jan, must be in the local format, case sensitive.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.