Unable to load csv file into elasticsearch using logstash

Elastic Stack Logstash
#1

hi, I'm using ubuntu as my os. My csv file is present in downloads folder( which i later moved it to opt/logstash folder) and my config file is placed in path etc/logstash/conf.d and is as follows:-

input {
file {
path => "/opt/logstash/test.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
}
}

output {
elasticsearch {

    action => "index"
    host => "localhost"
    index => "stock"
}
stdout {}

}
This file is saved as test-config.conf file. When i run the sudo service logstash configtest it return with Config Ok and then i restart the logstash service. After this i go to the opt/logstash path and run the command "bin/logstash -f /etc/logstash/conf.d/test-config.conf -v " It says no file to upload. I'm a newbie in elk please advise correct steps to ensure that the csv file is loaded. Also i'm not sure from which path we have to run the above command from ?..Any help will be greatly appreciated.

#2

After this i go to the opt/logstash path and run the command "bin/logstash -f /etc/logstash/conf.d/test-config.conf -v

You're running this command but you also run Logstash as a service?

It says no file to upload.

What's the exact message you get?

Also i'm not sure from which path we have to run the above command from ?

Doesn't matter as long as all paths are absolute.

#3

Magnus..thanks for the reply. My goal is to ensure my config file is correct and the end result is to have elasticsearch store the data coming froma csv file( via logstash). To answer your 1st question, No i am not running it as a service..I was trying to test my config file and i guess i had the wrong statement there..I just found out we should instead use "java -jar /opt/logstash/logstash.jar agent --configtest --config /etc/logstash/conf.d/test-config.conf" to test the config file, but i have an error...I have pasted the exact message and the commands below for reference:-

karthik@karthikvalluri:~$ sudo -i
[sudo] password for karthik:
root@karthikvalluri:~# sudo nano /etc/logstash/conf.d/test-config.conf
root@karthikvalluri:~# java -jar /opt/logstash/logstash.jar agent --configtest --config /etc/logstash/conf.d/test-config.conf
Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (InvalidOption) invalid option: --configtest
at RUBY.complete(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1542)
at org.jruby.RubyKernel.catch(org/jruby/RubyKernel.java:1174)
at RUBY.complete(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1540)
at RUBY.parse_in_order(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1354)
at org.jruby.RubyKernel.catch(org/jruby/RubyKernel.java:1174)
at RUBY.parse_in_order(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1347)
at RUBY.order!(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1341)
at RUBY.permute!(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1432)
at RUBY.parse!(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1453)
at RUBY.parse(jar:file:/opt/logstash/logstash.jar!/META-INF/jruby.home/lib/ruby/1.9/optparse.rb:1443)
at RUBY.parse_options(file:/opt/logstash/logstash.jar!/logstash/agent.rb:196)
at RUBY.run(file:/opt/logstash/logstash.jar!/logstash/agent.rb:320)
at logstash.runner.run(logstash/runner.rb:98)
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:249)
at logstash.runner.run(logstash/runner.rb:173)
at logstash.runner.main(logstash/runner.rb:71)
at logstash.runner.(root)(logstash/runner.rb:198)

#4

Just made a correction in my command "java -jar /opt/logstash/logstash.jar agent --config /etc/logstash/conf.d/test-config.conf"..There seems to be an error in my config file. the error msg is as follows:-
root@karthikvalluri:~# java -jar /opt/logstash/logstash.jar agent --config /etc/logstash/conf.d/test-config.conf
Missing required parameter 'type' for input/file {:level=>:error}
Config validation failed. {:level=>:error}
Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:842)
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:813)
at RUBY.config_init(file:/opt/logstash/logstash.jar!/logstash/config/mixin.rb:49)
at RUBY.initialize(file:/opt/logstash/logstash.jar!/logstash/inputs/base.rb:62)
at RUBY.initialize(file:/opt/logstash/logstash.jar!/logstash/inputs/file.rb:62)
at RUBY.parse_config(file:/opt/logstash/logstash.jar!/logstash/agent.rb:295)
at RUBY.parse(file:/opt/logstash/logstash.jar!/logstash/config/file.rb:54)
at RUBY.each(file:/opt/logstash/logstash.jar!/logstash/config/file.rb:92)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1612)
at RUBY.each(file:/opt/logstash/logstash.jar!/logstash/config/file.rb:91)
at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1257)
at RUBY.each(file:/opt/logstash/logstash.jar!/logstash/config/file.rb:84)
at RUBY.parse(file:/opt/logstash/logstash.jar!/logstash/config/file.rb:40)
at RUBY.parse_config(file:/opt/logstash/logstash.jar!/logstash/agent.rb:285)
at RUBY.run_with_config(file:/opt/logstash/logstash.jar!/logstash/agent.rb:387)
at org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)
at RUBY.run_with_config(file:/opt/logstash/logstash.jar!/logstash/agent.rb:386)
at RUBY.run(file:/opt/logstash/logstash.jar!/logstash/agent.rb:335)

#5

i put my config file in logstash folder and ran the config file ..the errors have certainly reduced.

root@karthikvalluri:/opt/logstash# sudo nano /opt/logstash/test-config.conf
root@karthikvalluri:/opt/logstash# bin/logstash -f test-config.conf
Settings: Default pipeline workers: 4
Error: The setting host in plugin elasticsearch is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask. {:level=>:error}

#6

I was looking at older posts and i just replaced host by hosts ..works great.

#7

I just found out we should instead use "java -jar /opt/logstash/logstash.jar agent --configtest --config /etc/logstash/conf.d/test-config.conf" to test the config file

Just use /opt/logstash/bin/logstash --configtest --config /etc/logstash/conf.d/test-config.conf.

#8