Autodetect_column_names is not working as expected in csv filter plugin


I have this logstash .conf file:

input {
  file {
    path => "/eee/*.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
filter {
  csv {
   autodetect_column_names => true

And multiple .csv files, containing different data, all of them need to be placed in the same index.
Example, file1.csv contains:


And file2.csv contains different headers:

And when I open index in Kibana, after indexing in elasticsearch I see that all the header names were taken from the first file
So, from file2.csv I see:
Name: Josh, City: 2020-04-05, Date: 2022-01-02, Comment: yes

How can I avoid this problem? Big thanks in advance. Using logstash 8.7.1, pipeline.workers is set to 1.

autodetect_column_names sets the names once and never changes them. If you want multiple sets of column names then you need multiple csv filters.

You may be able to use a conditional based on the number of fields, or else based on the [log][file][path].


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.