Extracting filename from log.file.path not working

Elastic Stack Logstash
1

I am trying to extract filename from log.file.path and then apply grok filter based on those as shown below. But the filename isnt being extracted. How can I extract the name and use the name to apply filters on?

filter {
  if[fields][tags] =="ngta-web" {
    grok {
      break_on_match => false
      match => {"log.file.path" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"}
      break_on_match => false
      if [filename] == "ngta-logs.txt"
      match => {
	"message" => [
	   "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{GREEDYDATA}\]%{SPACE}%{LOGLEVEL:log}%{SPACE}%{GREEDYDATA:logmessage}"
         ]
      }
    }
  }
else if [fields][tags] == "ngta-app" {
    grok {
      match => {"log.file.path" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"}
      break_on_match => false
      if [filename] == "ob-logs.txt"
      break_on_match => false
      match => { 
	"message" => [
	   "\A%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA} %{LOGLEVEL:loglevel} %{DATA:message} - MONITORING_APP_URL: %{URI:monitor_url} -> alertInstancesRootURL: %{URI:alert_url}" 
 	]
      }
    }
  }
2

If you are using filebeat I would expect you to have a field called [log][file][path] rather than [log.file.path] (with periods in the field name).

3

@badger I havent added log.file.path in filebeat as I thought filebeat adds log.file.path as metadata. Would [log][file][path] be added like tag too?

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true
  input_type: log
  fields:
    tags: ["obapp-dotnet"]

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt'
    - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt'
    # - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\log.txt'
    # - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\exceptionlog.txt'    
    output.logstash:
  # The Logstash hosts
  hosts: ["192.168.xx.xx:5044"]
  #hosts: ["192.168.xx.xx:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

# processors:
  # - add_host_metadata: ~
  # - add_cloud_metadata: ~
  # - add_docker_metadata: ~
  # - add_kubernetes_metadata: ~

#================================= Logging =======================================
logging.level: info
logging.to_files: true
logging.files:
  path: D:\filebeat\logs
  name: filebeat.log
  keepfiles: 7
4

filebeat does not add a field called [log.file.path]. It add a field called log, which is an object that contains a field called file, which contains a field called path. In logstash you have to refer to that as [log][file][path].

5

So something like this in logstash?

Do i add [log][file][path] to filebeat.yml too?

filter {
  if[fields][tags] =="ngta-web" {
    grok {
      break_on_match => false
      match => {"[log][file][path]" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"}
      break_on_match => false
      if [log][file][path] == "ngta-logs.txt"
      match => {
	"message" => [
	   "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{GREEDYDATA}\]%{SPACE}%{LOGLEVEL:log}%{SPACE}%{GREEDYDATA:logmessage}"
         ]
      }
    }
  }
1 Like
6

That looks better, yes.

7

But as I passed this, I have configuration error in the grok. Can i not have multiple matches inside grok?@Badger

filter {
   if [fields][log_type] == "obapp-dotnet" {
    grok {
      break_on_match => false
      match => {"[log][file][path]" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"}
      if [log][file][path] == "log2.log"{
      match => { 
	"message" => [
	   "%{DATESTAMP:timestamp}%{SPACE}%{NONNEGINT:code}%{GREEDYDATA}%{LOGLEVEL}%{SPACE}%{NONNEGINT:anum}%{SPACE}%{GREEDYDATA:logmessage}" 
 	]
      }
	}
    }
  }

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:test, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 13, column 10 (byte 255) after filter {\n   if [fields][log_type] == \"obapp-dotnet\" {\n    grok {\n      break_on_match => false\n      match => {\"[log][file][path]\" => \"%{GREEDYDATA}/%{GREEDYDATA:filename}\\.log\"}\n      if ", :backtrace=>["/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
9

Both the matches work on. I have filename, and timestamp, code, anum, logmessage as fields.

But the if line I have below doesnt work. So how can I add the conditional? SHould this if statemnet be outside the if [fields][log_type]?

filter {
   if [fields][log_type] == "obapp-dotnet" {
    grok {
      #break_on_match => true
      match => {"[log][file][path]" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"}
    }
      #if [log][file][path] == "log2.log"{
      grok{
        match => { 
	"message" => [
	   "%{DATESTAMP:timestamp}%{SPACE}%{NONNEGINT:code}%{GREEDYDATA}%{LOGLEVEL:loglevel}%{SPACE}%{NONNEGINT:anum}%{SPACE}%{GREEDYDATA:logmessage}" 
 	]
       }
      }
    #}
  }
10

This is a strict string equality test. If the path contains a directory it will not test true. Perhaps you want

if [log][file][path] =~ /log2.log$/ {
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.