Bulk export of several directories of csv files to elasticsearch

Elastic Stack Logstash
#1

I have several directories and each has several subdirectories and each has csv files within them. I would like to export all these csv files inside all directories to elasticsearch using logstash and automatically take column names. Note that different csv files have different column names. There are about 7 types of csv files in these directories and each type has the same column names.

Please suggest me what to do in such case.

input {
file {
path => "/dir/dir*///*.csv"
start_position => "beginning"
}
}

filter {
if ([message] =~ " ") {
drop { }
} else {
csv { }
}
}

output {
elasticsearch {
hosts => ["http://10.0.0.4:9200"]
}
stdout{}
}

#2

The csv filter has an autodetect_column_names option that probably does what you're looking for.

#3

But nothing in the documentation about how to use it.
Actually the columns are in the second row, not in the first row. What should I do in such case.

#4

But nothing in the documentation about how to use it.

Well, there's this: https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-autogenerate_column_names

Actually the columns are in the second row, not in the first row. What should I do in such case.

That's probably not supported.

#5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.