Logstash and a CSV

[OpSecMonkey]

Good Sunday Morning,

I have a question. I have a directory full of CSV files roughly 110 gigs or so. I am trying to upload them all to ES but I keep getting an error. I know ES is working because I am using a tool to upload 1 CSV at a time. It takes forever. I am on a Windows Server.

So this is what I have as a config file

input {
  file {
  path => "Z:\output\not\*.csv"
  start position => "beginning"
  sincedb_path => "/dev/null"
  }
}
filter {
  csv {
  separator => ","
  }
}
output {
  elasticsearch {
  host => "http://localhost:9200"
  index => "work"
  }
 stddout {}
}

Once I run that I got a error of this. I am not sure what is going on here.

Any help will be great.

[AquaX (Andreas Helmer)]

@OpSecMonkey

sincedb_path => "/dev/null"

This does not exist in windows
Change it to NUL

sincedb_path => "NUL"

[OpSecMonkey]

Thank you

[OpSecMonkey]

I get this error now and not sure why

[AquaX (Andreas Helmer)]

Did something change in your configuration?
It says that "since_path" is an invalid parameter... it should be "sincedb_path".

[OpSecMonkey]

Yes I just had the word "since_path" I changed it.

[balumurari1]

try like this,

input {
file {
path => "D:/Balu/ELK-stack/csvfile.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["open","high","low","close","volume"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "wallet-address-index"
}
stdout { codec => rubydebug }
}