Import CSV data to Elastic App Search "block in start_input" Error

Elastic Stack Logstash
1

I'm trying to import CSV data to my Elastic App search but it throwing an error.

Error Message:-

[2020-06-21T01:40:17,552][INFO ][filewatch.observingtail  ][main][07a10415268d8655d99cdad82a61cbca125e21687e65bc0296c88f92c9d9c4d1] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-21T01:40:17,555][ERROR][logstash.javapipeline    ][main][07a10415268d8655d99cdad82a61cbca125e21687e65bc0296c88f92c9d9c4d1] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["/Users/thehopper/Documents/elastic/Raw_Data/first.csv"], id=>"07a10415268d8655d99cdad82a61cbca125e21687e65bc0296c88f92c9d9c4d1", sincedb_path=>"/dev/null", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_a55d2b4b-0837-4754-88e1-ee7ca0dc916f", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false, check_archive_validity=>false>
  Error: Permission denied - Permission denied
  Exception: Errno::EACCES
  Stack: org/jruby/RubyFile.java:1267:in `utime'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1133:in `block in touch'
org/jruby/RubyArray.java:1809:in `each'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1130:in `touch'
/Users/thehopper/Documents/elastic/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/sincedb_collection.rb:22:in `initialize'
/Users/thehopper/Documents/elastic/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/observing_base.rb:62:in `build_watch_and_dependencies'
/Users/thehopper/Documents/elastic/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/filewatch/observing_base.rb:56:in `initialize'
/Users/thehopper/Documents/elastic/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/logstash/inputs/file.rb:345:in `start_processing'
/Users/thehopper/Documents/elastic/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.18/lib/logstash/inputs/file.rb:361:in `run'
/Users/thehopper/Documents/elastic/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:345:in `inputworker'
/Users/thehopper/Documents/elastic/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:336:in `block in start_input'

Config file:-

input {
  file {
    path => ["/Users/thehopper/Documents/elastic/Raw_Data/first.csv"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["first","last","Full Name","Solved*","Age","Gender","Race","Date","Address","Neighborhood","status","lat","lon","Map Description","Story","URL"]
  }
}
output {
   elastic_app_search {
     url => "http://localhost:3002"
     api_key => "private-key...."
     engine => "email-pass-engine"
  }
  stdout {
    codec => rubydebug
  }
}
2

That code is trying to touch the sincedb. It appears you do not have write access to /dev/null. Check the permissions and ownership of it.

3

Thank you for your response. You were right, I run the command with Sudo now it's working.

4

can you help me with another thing, my import file so big. Is there any way to see the progress of the import.

I have tried with small file but after finish import, it's not showing any successful mange So If my file is big how do I know it's or not?

5

The pipeline stats API can be used to see how many events have passed through input, filters, and outputs.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.