Extract time from txt file and date from file name and add them to @timestamp field

Elastic Stack Logstash
1

I am trying to extract time field from txt file which looks like:
14:39:38 someName clientName 968112300 1008008000 39895700
also tyring to extract date from file name /path/to/file/name/filename_20220727.txt
finally, concatenate both time and date then add both field values to @timestamp field.

my logstash conf file looks like:

input {
  file {
  path => "/path/to/file/name/filename_*.txt"
  start_position => beginning
  sincedb_path => "/dev/null"
  }
}
filter {
        grok {
            patterns_dir => ["/etc/logstash/conf.d/patterns"]
            match => { "message" => [ "%{TIME:time}\s*%{USERNAME:User}\s*%{USER:ident}\s*%{NUMBER:AccountBef}\s*%{NUMBER:AccountAft}\s*%{NUMBER:ReloadedAmount}" ] }
        }
        grok {
          match => ["path", "(/%{GREEDYDATA}/balanceReload_%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}/.txt$)"]
          add_field => ["nDate", "%{year}/%{month}%/{day} %{TIME}"]
        }
       date {
            match => ["nDate", "yyyy-MM-dd HH:mm:ss"]
            target => "@timestamp"
            timezone => "UTC"
        }

}
output {
  stdout {}
}

the @timestamp field value is still showing the document date rather than in date and time values of newly added field nDate Besides, I get _grokparsefailure

    "event" => {
        "original" => "16:30:10   somename        clientName            813954800     954128700      140173900    "
    },
              "User" => "somename",
    "ReloadedAmount" => "140173900",
        "AccountBef" => "813954800",
           "message" => "16:30:10   somename        clientName            813954800     954128700      140173900    ",
               "log" => {
        "file" => {
            "path" => "/path/to/file/name/filename__20220725.txt"
        }
    },
              "tags" => [
        [0] "_grokparsefailure"
    ],
        "@timestamp" => 2022-08-02T09:23:06.915422Z,
              "time" => "16:30:10",
        "AccountAft" => "954128700",
          "@version" => "1"
}
{
             "ident" => "clientName",
              "host" => {
        "name" => "hostName"
    },
2

The grok that tries to match [path] fails because you do not have a [path] field, you have a [log][file][path] field. Since the grok does not match the add_field is not executed.

Once you fix the filename the grok will still fail because "filename" does not match "balanceReload". Once you fix that it could still fail because %{YEAR} may consume 4 digits, leaving nothing for %{MONTHDAY} to match.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.