Can't load CSV data into elasticsearch index


(elk_enthu) #1

Hi there,
Since last 3 days, I’m trying to explore ELK stack and importing CSV table into Elasticsearch through Logstash. Checked many forums and videos and tried the following. But, I get various errors. Here’s what I’ve done so far:

OS: Windows 10
ELK version: 6.4.2

  • Successfully started ElasticSearch by running
    PATH\bin\elasticsearch.bat

  • Successfully started Kibana by running
    PATH\bin\kibana.bat

  • Successfully started the Logstash pipe by running
    bin\logstash -e "input { stdin { } } output { stdout {} }" I learnt that for Windows "double quotes" required instead of single.

  • On Kibana I created an index by running the following on Dev tools console

      PUT /customer/
      {
       "mappings":{
       "customer":{
       "properties":{
       "LMK_KEY":{ "type": "long" },
       "POSTCODE" : { "type" : "text" },
       "C_ENG_RATING" : { "type" : "text" },
       "P_ENG_RATING" : { "type" : "text" },
       "C_ENG_EFF" : { "type" : "integer" },
       "P_ENG_EFF" : { "type" : "integer" },
       "PROPERTY_TYPE" : { "type" : "text" },
       "BUILT_FORM" : { "type" : "text" }
       }
       }
       }
      }
    

I can see the newly created index on Index Management section of ES.

Created the following config file.

input {
  file {
    path => "PATH/ELK_stack/epc.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["LMK_KEY", "POSTCODE", "C_ENG_RATING", "P_ENG_RATING", "C_ENG_EFF", "P_ENG_EFF", "PROPERTY_TYPE", "BUILT_FORM"]
  }
  }
output {
   elasticsearch {
     hosts => "localhost:9200"
     index => "customer"
     document_type => "customer"
  	}
	 stdout {codec => rubydebug}
}

The config file saved in,

PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\config\logstash_epc.config

If I run the config file in a new cmd window, I get the following error.

E:\PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\bin>logstash -f E:/PATH/ELK_stack/logstash-6.4.2/logstash-6.4.2/config/logstash_epc.config
Sending Logstash logs to E:/PATH/ELK_stack/logstash-6.4.2/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-10-18T16:58:51,031][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-10-18T16:58:51,063][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[2018-10-18T16:58:51,073][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

E:\PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\bin>

I terminated the pipeline by CTRL+C > Y. and then re-run

E:\PATH\ELK_stack\logstash-6.4.2\logstash-6.4.2\bin>logstash -f E:/PATH/ELK_stack/logstash-6.4.2/logstash-6.4.2/config/logstash_epc.config

My cmd window just hangs up with following error and goes on, just updating time

E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:56:in `initialize'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:332:in `start_processing'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337:in `run'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[2018-10-18T17:43:38,668][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2018-10-18T17:43:38,671][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["E:/GIS_miscel/ELK_stack/epc.csv"], id=>"3cbe707b339631942274e643484f98990e7ae77eae45a0d005e216a89a975f71", sincedb_path=>"/dev/null", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_4dfa964b-02b8-4794-9324-73f2d0fff191", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc">
  Error: No such file or directory - E:/dev/null
  Exception: Errno::ENOENT
  Stack: org/jruby/RubyFile.java:366:in `initialize'
org/jruby/RubyIO.java:1154:in `open'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1167:in `block in touch'
org/jruby/RubyArray.java:1734:in `each'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1161:in `touch'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/sincedb_collection.rb:22:in `initialize'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:62:in `build_watch_and_dependencies'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:56:in `initialize'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:332:in `start_processing'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337:in `run'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
E:/GIS_miscel/ELK_stack/logstash-6.4.2/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'

Can someone please give pointers? Thanks in advance.


#2

follow this

hope this helps


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.