Logstash Input-Path as Index

[Robsen_Inc]

Hi Community,

I would like to use the Path as an index.

The question has been asked many times in different forums, but unfortunately the solutions presented do not work for me.

Maybe you can help me.
Specifically, I want to read in my entire folder of records with:
path => "/logs/inputdata/cardets/*"
and then find in Elasticsearch the indexes related to the files. like "ta1-cadets-e3-official-1.json.4".

attached my (now hard-coded) pipeline config:

input{
  file {
     path => "/logs/inputdata/cardets/ta1-cadets-e3-official-1.json.4"
     start_position => "beginning" 
     codec => "json"
     type => cardets
     sincedb_path => "/dev/null"
  }
}
filter {
  mutate {
    add_field => {
      "event.dataset" => "cadets"
    }
  }
}
output {
  elasticsearch {
    hosts => "${ELASTICSEARCH_HOSTS}"
    user => "${ELASTICSEARCH_USERNAME}"
    password => "${ELASTIC_PASSWORD}"
    index => "ta1-cadets-e3-official-1.json.4"
    ssl => true
    cacert => '/usr/share/logstash/config/certs/ca/ca.crt' 
  }
}

[Badger]

In recent versions the index option to the Elasticsearch output is ignored because ILM is enabled.

If you disable that you could use index => "%{[log][file][path]}" (assuming ECS is enabled), although using a large number of small indexes is very inefficient.