Hi there,
Since last 3 days, I’m trying to explore ELK stack and importing CSV table into Elasticsearch through Logstash. Checked many forums and videos and tried the following. But, I get various errors. Here’s what I’ve done so far:
OS: Windows 10
ELK version: 6.4.2
-
Successfully started ElasticSearch by running
PATH\bin\elasticsearch.bat -
Successfully started Kibana by running
PATH\bin\kibana.bat -
Successfully started the Logstash pipe by running
bin\logstash -e "input { stdin { } } output { stdout {} }"I learnt that for Windows "double quotes" required instead of single. -
On Kibana I created an index by running the following on Dev tools console
PUT /customer/ { "mappings":{ "customer":{ "properties":{ "LMK_KEY":{ "type": "long" }, "POSTCODE" : { "type" : "text" }, "C_ENG_RATING" : { "type" : "text" }, "P_ENG_RATING" : { "type" : "text" }, "C_ENG_EFF" : { "type" : "integer" }, "P_ENG_EFF" : { "type" : "integer" }, "PROPERTY_TYPE" : { "type" : "text" }, "BUILT_FORM" : { "type" : "text" } } } } }
I can see the newly created index on Index Management section of ES.
Created the following config file.
input {
file {
path => "PATH/ELK_stack/epc.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["LMK_KEY", "POSTCODE", "C_ENG_RATING", "P_ENG_RATING", "C_ENG_EFF", "P_ENG_EFF", "PROPERTY_TYPE", "BUILT_FORM"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "customer"
document_type => "customer"
}
stdout {codec => rubydebug}
}
The config file saved in,
PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\config\logstash_epc.config
If I run the config file in a new cmd window, I get the following error.
E:\PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\bin>logstash -f E:/PATH/ELK_stack/logstash-6.4.2/logstash-6.4.2/config/logstash_epc.config
Sending Logstash logs to E:/PATH/ELK_stack/logstash-6.4.2/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-10-18T16:58:51,031][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-10-18T16:58:51,063][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2018-10-18T16:58:51,073][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
E:\PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\bin>
I terminated the pipeline by CTRL+C > Y. and then re-run
E:\PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\bin>logstash -f E:/PATH/ELK_stack/logstash-6.4.2/logstash-6.4.2/config/logstash_epc.config
My cmd window just hangs up with following error and goes on, just updating time
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:56:in `initialize'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:332:in `start_processing'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337:in `run'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[2018-10-18T17:43:38,668][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2018-10-18T17:43:38,671][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["E:/GIS_miscel/ELK_stack/epc.csv"], id=>"3cbe707b339631942274e643484f98990e7ae77eae45a0d005e216a89a975f71", sincedb_path=>"/dev/null", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_4dfa964b-02b8-4794-9324-73f2d0fff191", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc">
Error: No such file or directory - E:/dev/null
Exception: Errno::ENOENT
Stack: org/jruby/RubyFile.java:366:in `initialize'
org/jruby/RubyIO.java:1154:in `open'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1167:in `block in touch'
org/jruby/RubyArray.java:1734:in `each'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1161:in `touch'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/sincedb_collection.rb:22:in `initialize'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:62:in `build_watch_and_dependencies'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:56:in `initialize'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:332:in `start_processing'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337:in `run'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
Can someone please give pointers? Thanks in advance.