Using logstash to import csv files into elasticsearch

Elastic Stack Logstash
1

I'm using Running Logstash on Docker | Logstash Reference [5.6] | Elastic to import csv file into elasticsearch.

following is content from my ./pipeline directory:

input {
    file {
        path => "/csv/in/*.csv"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
  csv {
    separator => ";"
    columns => ["bin", "cardNetwork", "bank", "cardType", "cardCategory", "country"]
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    user => "USER"
    password => "PASSWORD"
    index => "INDEX"
    document_type => "csv"
  }
  stdout { codec => rubydebug }
}

I'm not seeing any "ERROR" even with 'log.level' set to 'debug', thanks to stdout part I'm able to see that csv parse correctly, yet I'm not seeing anything written to elasticsearch.

Please advise.

2

I'm not seeing anything written to elasticsearch.

What made you reach that conclusion? How are you checking what's in ES?

3

I checked using Kibana and/or curl for newly created _index: INDEX, yet no records in there...

4

I assume the index isn't really named INDEX? Because that would be an illegal name.

If the index exists then Logstash is at least able to talk to ES (unless you created the index yourself). If there really isn't anything in the log I'd trace the network traffic.

5

@magnusbaeck, you're correct INDEX isn't real...

... to test your theory, I adjusted output to write to index that doesn't exists followed by restart of container, and as last time I see stdout (rubydebug), however new index was not created nor my data would appear inside of it... I look in logs and there is no errors (or any issues) with connecting to elasticsearch, it appears to be connected just fine.

I'm also using X-Pack, and I see other connection to elasticsearch (for monitoring) are made without any issues...

Please advise.

6

As I said, I'd trace the network traffic so see exactly what's going on.

7

@magnusbaeck, I'm on same network as elasticsearch, network isn't preventing any communications as I mention X-Pack's monitoring part is working without any issues and it's using same elasticsearch as I specified in my output plugin.

8

The point of inspecting the network traffic is to see whether Logstash is attempting to send anything to ES, and if so what and what ES responds. Any errors response should show up in the Logstash log but other breakage could prevent that from happening.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.