Logstash and a CSV


(OpSec Monkey) #1

Good Sunday Morning,

I have a question. I have a directory full of CSV files roughly 110 gigs or so. I am trying to upload them all to ES but I keep getting an error. I know ES is working because I am using a tool to upload 1 CSV at a time. It takes forever. I am on a Windows Server.

So this is what I have as a config file

input {
  file {
  path => "Z:\output\not\*.csv"
  start position => "beginning"
  sincedb_path => "/dev/null"
  }
}
filter {
  csv {
  separator => ","
  }
}
output {
  elasticsearch {
  host => "http://localhost:9200"
  index => "work"
  }
 stddout {}
}

Once I run that I got a error of this. I am not sure what is going on here.

Any help will be great.


(Andreas H) #2

This does not exist in windows :slight_smile:
Change it to NUL

sincedb_path => "NUL"


(OpSec Monkey) #3

Thank you :slight_smile:


(OpSec Monkey) #4

I get this error now and not sure why


(Andreas H) #5

Did something change in your configuration?
It says that "since_path" is an invalid parameter... it should be "sincedb_path".


(OpSec Monkey) #6

Yes I just had the word "since_path" I changed it.


#7

try like this,

input {
file {
path => "D:/Balu/ELK-stack/csvfile.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["open","high","low","close","volume"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "wallet-address-index"
}
stdout { codec => rubydebug }
}