Preserve original log file name using Filebeat and Logstash

Hi. I just want to preserve the original file name for example: laravel-2022-05-17.log and set it as index.

I was not able to find the propper way to obtain it using filebeat and sending it to logstash.

Thanks :slight_smile:

Hi, solved. For each file I wanted to have a custom index name I had to define a different log. My two log files are userimports.log and push-notifications.log

Filebeat.yml:

filebeat.inputs:
# Laravel Logs
- type: log
  enabled: true
  paths:
    - /var/log/dmesg_log/userimports/push-notifications.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  fields:
    logType: "laravel"
  tags: ["push-notifications"]

- type: log
  enabled: true
  paths:
    - /var/log/dmesg_log/userimports/userimports.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  fields:
    logType: "laravel"
  tags: ["userimports"]


  # Enrich laravel logs with docker data (not working)
  # processors:
  # - add_docker_metadata:
  #     host: "unix:///var/run/docker.sock"

# setup.kibana:
#   host: "http://elasticsearch:5601"

output.logstash:
  hosts: ["logstash:5044"]

And now using the tags we can determine which log is which in logstash.conf:

input {
  beats {
    port => 5044
  }
}

##filter data filtering operation
filter {
  grok {
    match => {
      "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: (?<log>[^{]+)?%{GREEDYDATA:raw-json}"
    }
  }

  grok {
    match => {
      "raw-json" => "(?<raw-process-json>\{(.*)\})%{GREEDYDATA:response}"
      tag_on_failure => [ ]
    }
  }

  json {
    source => "raw-process-json"
    target => "json"
  }

  mutate {
    rename => { "message" => "raw-message" }
    rename => { "json" => "raw-process-json" }
  }
}

output {
    if "push-notifications" in [tags] {    
        elasticsearch{
          hosts => ["http://elasticsearch:9200"]
          index => "push-notifications-%{+YYYY.MM.dd}"
        }
        stdout {}
    }
    if "userimports" in [tags] { 
        elasticsearch{
          hosts => ["http://elasticsearch:9200"]
          index => "userimports-%{+YYYY.MM.dd}"
        }
        stdout {}
    }
}

Hope it helps!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.