Can I get the current path of file being processed by Logstash file input when searching through subdirectories?

Elastic Stack Logstash
1

Hey,

I'm wondering if there is anyway to tell what subdirectory/path to file is currently getting processed by Logstash when using the file input to look through a directory tree?

From what I understand I could parse my subdirectories in two ways given my setup, either using a recursive glob /path/to/top/directory/**/*.log or using a direct wildcard path: /path/to/top/directory/*/*/*.log

Either way would be fine for me but what I really want is to know the current path that logstash is operating on. Essentially the subdirectory tells me a name of a server like this: /path/to/top/directory/server1/2018_03_21/foo.log

I then want to add a field with that server number in the file input block: add_field => {"server", "1"}

Is this possible using logstash? Is there someway I can access the path that is currently being processed? If I could get the path I could then use grok to parse the server number into a variable and use that in add_field

Also, unfortunately I can't use filebeat in my setup.

Thanks!
Dave

2

There should be a path field in the final event, containing the full path (including filename), basically what you're after.

1 Like
3

Yes, path was exactly what I needed, thanks @paz! My final solution ended up being:

input {
    file {
       path => "/path/to/top/directory/*/*/*.log"
     }
}

filter {
  grok {
      match => ["path", "^/path/to/top/directory/server(?<server>[^/]+)/"]
  }
}

Note that grok automatically adds named captures as a field so there is no need for add_field

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.