Logstash Conf | Extracting Filename from Path

Elastic Stack Logstash
1

I am trying to setup Logstash to feed Elasticsearch. In course, I've created the following conf file that seem to work nicely:

input {
  beats {
    port => 5044
  }
  
  file {
    path => "C:/f1/f2/Logs/f3/LocalHost#base#iway_2022-03-28T10_45_15.log"
  }
}

filter {
  grok {
    match => {
      "message" => [
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .(W.)%{DATA:thread}.%{INT:thread_pool}. %{GREEDYDATA:msgbody}",
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .%{DATA:thread}. %{GREEDYDATA:msgbody}"      
      ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "iway_logs"
    user => "elastic"
    password => "something"
    cacert => "C:\f1\f2\logstash-8.1.3\config\cert\elasticsearch_http_ca.crt"
  }
}

I have been trying to add two new fields but unsuccessful so far. Following is the current version of the conf file after several revises.

input {
  beats {
    port => 5044
  }
  
  file {
    path => "C:/f1/f2/Logs/f3/LocalHost#base#iway_2022-03-28T10_45_15.log"
  }
}

filter {
  grok {
    match => {
      "message" => [
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .(W.)%{DATA:thread}.%{INT:thread_pool}. %{GREEDYDATA:msgbody}",
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .%{DATA:thread}. %{GREEDYDATA:msgbody}"      
      ]
    }
  }
  grok {
        match => { 
            "path" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"
            }
  }
  mutate {
        split => { "filename" => "#" }
        add_field => { "serverName" => "%{[filename][0]}" }
        add_field => { "configName" => "%{[filename][1]}" }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "iway_logs"
    user => "elastic"
    password => "something"
    cacert => "C:\f1\f2\logstash-8.1.3\config\cert\elasticsearch_http_ca.crt"
  }
}

The result of new fields namely, serverName and configName, always reports the raw expression as opposed to an evaluated output. Could someone help? TIA.

2

That is telling you that the [filename] field does not exist. Do you really have a [path] field? With ECS (enabled by default now) I would not expect that field to be called that.

3

That was indeed the case. I had to apply the complete context path of the field, that is, [log][file][path]. Thanks for your feedback.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.