Import (21gb) csv to elasticsearch

Elastic Stack Elasticsearch
2

Hi @examin,

Logstash has a csv filter.

You could start with a few lines and just output to STDOUT.

The Logstash config would be something like (I have not used this so might not be 100% right)

input { stdin { } }

filter {
  csv {
    autodetect_column_names => true
  }
}
output {
  stdout { codec => rubydebug }
}

Download Logstash and start with

logstash-6.3.1/bin/logstash -f above_config.conf

Paste in one line and see how it is parsed. Once you are happy you can change input to file and output to Elasticsearch.

I'm sure there are many other ways to do it as well.

1 Like