Help with dissect in filter for logstash.conf to dynamically append filename to index patternNotFound Error

Elastic Stack Logstash
1

My folder structure is as follows:

main_directory.
|   .env
|   docker-compose.yml
|
+---elasticsearch
|   \---config
|           elasticsearch.yml
|
\---logstash
    +---config
    |   |   logstash.yml
    |   |   pipelines.yml
    |   |   
    |   \---pipeline
    |           logstash.conf
    |
    +---data
    |   |   .lock
    |   |   dummy.csv
    |   |   test.csv
    |   |   uuid
    |   |
    |   +---dead_letter_queue
    |   \---queue
    \---logs

My logstash.conf is as follows:

input {
  file {
    path => "/usr/share/logstash/data/*.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  csv {
    separator => ","
    # columns => ["User_Id","First_Name"]
    autodetect_column_names => true
    skip_header => true
  }
  dissect{
    mapping => {
      "path" => "/usr/share/logstash/data/#%{file_name}.csv"

    }
  }
}

output {
  elasticsearch {
    hosts => "${ELASTICSEARCH_HOST}"
    user => "${ELASTICSEARCH_USERNAME}"
    password => "${ELASTICSEARCH_PASSWORD}"
    index => "my_index_prime_%{file_name}"
    ssl => true
    ssl_certificate_verification => false
    manage_template => false
  }
}

Inside the /data/ directory there are many csv like test.csv, sample.csv, work_index_one.csv etc.

I used dissect in filter to add filenames to my index in the format of my_index_prime_${file_name} but I got the patternNotFound error:

[2023-05-17T19:38:16,515][WARN ][org.logstash.dissect.Dissector][main][3b3f27d0e8ab71d66a5d2b9d3b16693ce141224b733810c11fea3544cb5beef7] Dissector mapping, pattern not found {"field"=>"path", "pattern"=>"/usr/share/logstash/data/#%{file_name}.csv", "event"=>{"tags"=>["_dissectfailure"], "@version"=>"1", "message"=>"002,Asif\r", "path"=>"/usr/share/logstash/data/dummy.csv", "First_Name"=>"Asif", "@timestamp"=>2023-05-17T19:38:16.308Z, "host"=>"f47d5a099289", "User_Id"=>"002"}}

Any advice would be highly appreciated!

I'm trying to dynamically append file name to my index

2

Your dissect mapping requires that the filename start with #, but the path field in your event does not contain #, so there is no match.

3

thank you!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.