Use part of the log files name for the index?

Elastic Stack Logstash
1

I have my files all in the same directory names like such:

/var/log/apache2/testsite1.com_error.log
/var/log/apache2/testsite1.com_access.log
/var/log/apache2/subdomain.testsite1.com_error.log
/var/log/apache2/subdomain.testsite1.com_access.log

What I would like to do is have each of those get grouped into there own index. For example:

apache-testsite1.com-2020-08-02
apache-subdomain.testsite1.com-2020-08-02

Any thoughts on how to get this done? Here is my config file:

input {
  file {
    path => "/logs/apache2/*.log"
    sincedb_path => "/logstash/data/sincedb-apache-access"
  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  } else if [path] =~ "error" {
    mutate { replace => { type => "apache_error" } }
  } else {
    mutate { replace => { type => "random_logs" } }
  }

  grok {
    match => { "path" => ["(?<domain>[a-zA-Z0-9-.]+(?=_))"] }
  }
}

output {
  elasticsearch { 
      hosts => ["elasticsearch:9200"]
      index => "apache-${domain}-%{+YYYY.MM.dd}"
  }
  # stdout { codec => rubydebug }
}
2

A thoughtless solution is to have 1 file input per each log type. Then you can tag it add use that tag in the name.

3

So I solved this with the following changes:

Grok Line:

match => { "path" => "(?<domain>[a-zA-Z0-9-.]+(?=_))" }

Output index:

index => "apache-%{domain}-%{+YYYY.MM.dd}"
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.