Logstash - Get index name from filename

Elastic Stack Logstash
1

Hello,
I am loading files by logstash and I would like to know if it possible to set index from filename.
For example my files are:

system1-yyyyMMdd.csv (system1-20200106.csv)
system2-yyyyMMdd.csv (system2-20200106.csv)

and I would like to create indexes with "system1" and "system2", so I need to separate the first part from the filename.

input {
  file {
    path => ["/usr/share/logstash/data1/*.csv"]
    start_position => "beginning"
  }
}
filter {
  csv {
    separator => ";"
    columns => ["datetime", "level", "statuscode", "message", "endpoint"]
  }
}
output {
  elasticsearch {
    hosts => ["http://host.docker.internal:9200"]
    index => "index"
  }
}
2

Hi,

Not with your current configuration. You could an input for each file and add a field that you can use in the index name.

input {
  file {
    path => ["/usr/share/logstash/data1/system1.csv"]
    start_position => "beginning"
    add_field => "system1"
  }
  file {
    path => ["/usr/share/logstash/data1/system2.csv"]
    start_position => "beginning"
    add_field => "system2"
  }
}
filter {
  csv {
    separator => ";"
    columns => ["datetime", "level", "statuscode", "message", "endpoint"]
  }
}

# Setup index name
filter {
    if [system1] {
        mutate { add_field => { "[@metadata][index_name]" => "system1-%{+YYYY.MM.dd}" } }
    } else if [system2] {
        mutate { add_field => { "[@metadata][index_name]" => "system2-%{+YYYY.MM.dd}" } }
    } else {
        mutate { add_field => { "[@metadata][index_name]" => "unknown-system-%{+YYYY.MM.dd}" } }
    }
}

output {
  elasticsearch {
    hosts => ["http://host.docker.internal:9200"]
    index => "%{[@metadata][index_name]}"
  }
}
3

The file input adds a path field to events. You want to extract everything between the last / in the path and the first - in that section, which is

grok { match => { "path" => "(?<[@metadata][filePrefix]>[^/\-]+)-[^/]+$" } }

then

 index => "%{[@metadata][filePrefix]}"
5

Thanks very much. It is working very well.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.