Updating an existing field using path data

Elastic Stack Logstash
1

Hello Guys,

I am using a path which is formed by /dir/subdir/filename_log.gz.

I want to extract the filename and update an existent Field called Hostname with this information.

I have tried to use this code, but I had no success.

grok {
        match => { "[log][file][path]" => "/dir/subdir/%{GREEDYDATA:node}_log"}
        }
    mutate {
       replace => { "Hostname" => "%{[node][0]}" }
    }

Can you help me please?

2

From the grok you shared, the node field is not an array, so you should just use %{node} in the mutate filter, not %{[node][0]}.

3

Hi Leandro. Thanks for answering. Actually I have already tried to use %{node} instead , however it did not work.The field Hostname was filled with the %{node} string.
Do you guys have another suggestion?
Thanks in advance.

4

This indicates that the node field does not exist in your document. Is your grok work? Do you have any _grokparsefailure tag in your document?

You need to provide more information, please share the output you are getting in Logstash, without it is not possible to know what may be the issue.

5

I just simulated your filters here and it worked for me.

I used the following pipeline to simulate:

#
input {
    generator {
      lines => [
        "/dir/subdir/filename_123_456_log.gz",
        "/dir/subdir/filename_log.gz",
        "/dir/subdir/file_name_log.gz"
      ]
      count => 1
    }
}
filter {
    grok {
        match => {
            "message" => "/dir/subdir/%{GREEDYDATA:node}_log"
        }
    }
    mutate {
        replace => {
            "Hostname" => "%{node}"
        }
    }
}
#
output {
    stdout {}
}

And this is the output:

{
       "message" => "/dir/subdir/filename_log.gz",
          "host" => "lab",
      "Hostname" => "filename",
    "@timestamp" => 2023-03-21T12:30:47.758Z,
          "node" => "filename",
      "sequence" => 0,
      "@version" => "1"
}
{
       "message" => "/dir/subdir/filename_123_456_log.gz",
          "host" => "lab",
      "Hostname" => "filename_123_456",
    "@timestamp" => 2023-03-21T12:30:47.737Z,
          "node" => "filename_123_456",
      "sequence" => 0,
      "@version" => "1"
}
{
       "message" => "/dir/subdir/file_name_log.gz",
          "host" => "lab",
      "Hostname" => "file_name",
    "@timestamp" => 2023-03-21T12:30:47.759Z,
          "node" => "file_name",
      "sequence" => 0,
      "@version" => "1"
}
6

Hi Leandro,

Thanks for supporting me. Now It is working. I was probably typing something wrong when I used %{node} to replace the field. An important note is that it only worked when I used [log][file][path] to grok the name of the file.Using message did not work for me.

By means of study, here is my code. No grok failure and Hostname was updated with the name of the file.

input {
    file {
        path => "/dir/subdir/filename_log_*.gz"
        mode => "read"
        codec => multiline {
            pattern => "^\"(north|south)\""
            negate => true
            what => "previous"
         }

        }
}

filter {
    mutate {
        gsub => [ "message", "\r", "" ]
    }

    csv {
         columns => ["X","Y","W","Z"]
         skip_header => true
    }
    mutate {
        add_field => { "Pod" => "%{Hostname}"} 
    }
    grok {
        match => { "[log][file][path]" => "/dir/subdir/%{GREEDYDATA:node}_log"}
        #match => { "message" => "/dir/subdir/%{GREEDYDATA:node}_log"}
    }
    mutate {
       replace => { "Hostname" => "%{node}" }
    }
 }

Have a nice day!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.