Error code when importing csv file with logstash

Elastic Stack Logstash
1

hi logstash team

I am trying to import csv file andI cant figure out why I am getting below error, your help would be greatly appreciated

./logstash -f ../conf/logstash.yml
Sending Logstash's logs to /home/d/Documents/logstash-6.2.4/logs which is now configured via log4j2.properties
[2018-10-02T21:32:03,882][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/d/Documents/logstash-6.2.4/modules/fb_apache/configuration"}
[2018-10-02T21:32:03,945][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/d/Documents/logstash-6.2.4/modules/netflow/configuration"}
[2018-10-02T21:32:04,055][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "" hasn't been registered>, :backtrace=>["/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/settings.rb:37:in `get_setting'", "/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/settings.rb:70:in `set_value'", "/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/settings.rb:89:in `block in merge'", "org/jruby/RubyHash.java:1343:in `each'", "/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/settings.rb:89:in `merge'", "/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/settings.rb:138:in `validate_all'", "/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/runner.rb:264:in `execute'", "/home/d/Documents/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/home/d/Documents/logstash-6.2.4/logstash-core/lib/logstash/runner.rb:219:in `run'", "/home/d/Documents/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/home/d/Documents/logstash-6.2.4/lib/bootstrap/environment.rb:67:in `<main>'"]}
[2018-10-02T21:32:04,073][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

my yml file looks like this

input {
  file {
    path => "/home/d/pcap/test.csv"
  }
}
filter {
  csv {
     separator => ","
    columns => ["No","Time","Source","Destination","Protocol","Length","Info"]
  }
}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "pcap"
  }
stdout {}
}
2

Try the following:

input {
  file {
    path => "/home/d/pcap/test.csv"
    start_position => "beginning"
    sincedb_path => "/var/log/tracking_no"
 }
  }

filter {
csv {
  columns => ["No","Time","Source","Destination","Protocol","Length","Info"]
  separator => ","
}
}


output {
stdout { codec => json }
elasticsearch {
hosts => ["localhost:9200"]
index => "pcap-%{+YYYY.MM.dd}"
}
}
1 Like
3

Thanks @bardie , I changed yml to the one you sugested but it doesn't work.

this is what I get

d@ubuntu:~/Documents/logstash-6.4.2/bin$ ./logstash -f ../config/logstash1.yml 
Sending Logstash logs to /home/d/Documents/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-10-04T09:37:18,170][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "" hasn't been registered>, :backtrace=>["/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/settings.rb:36:in `get_setting'", "/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/settings.rb:69:in `set_value'", "/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/settings.rb:88:in `block in merge'", "org/jruby/RubyHash.java:1343:in `each'", "/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/settings.rb:88:in `merge'", "/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/settings.rb:137:in `validate_all'", "/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/runner.rb:237:in `run'", "/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/home/d/Documents/logstash-6.4.2/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-10-04T09:37:18,187][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
d@ubuntu:~/Documents/logstash-6.4.2/bin$

has anyone had this problem before? yml file looks legit

4

Please could you post your logstash.yml file

5

sure, its logstash1.yml but i am calling it with -f when trying to run logstash (dont think this is the problem)

d@ubuntu:~/Documents/logstash-6.4.2/bin$ cat ../config/logstash1.yml 
input {
  file {
    path => "/home/d/pcap/test.csv"
    start_position => "beginning"
    sincedb_path => "/var/log/tracking_no"
 }
  }

filter {
csv {
  columns => ["No","Time","Source","Destination","Protocol","Length","Info"]
  separator => ","
}
}


output {
stdout { codec => json }
elasticsearch {
hosts => ["localhost:9200"]
index => "pcap-%{+YYYY.MM.dd}"
}
}
d@ubuntu:~/Documents/logstash-6.4.2/bin$
6

Logstash has one configuration file called logstash.yml

Sample:

This file determines how logstash runs. If you do not have this file then that might be whats causing the error.

The file that you have posted is the logstash parser.

This is my logstash directory. The conf.d folder contains all my logstash parsers and the logstash.yml contains the logstash configurations.

image

Do you have a logstash.yml file other than the one you submitted?

7

hi there,

I have this file and it contains exactly what I shared before - just to be sure it is not the case I renamed it once again to logstash.yml but still error message remains the same.

I seen in other posts that the problem might be with logstash.yml content but I dont see there anything what could cause it.

it looks like I got all files where they should be in config folder as well

d@ubuntu:~/Documents/logstash-6.4.2/config$ ls
jvm.options  log4j2.properties  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options
d@ubuntu:~/Documents/logstash-6.4.2/config$

I wasnt changing there anything except that logstash.yml

8

STEP 1: Create a folder called pipeline
STEP 2: Rename the logstash.yml to sample.conf and move it to the pipelines folder
STEP 3: Create a new file called logstash.yml and paste the following

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline

Then try running it

9

ok this changed output and I dont see critical error at the moment but that last message is being constantly thrown on the console. no index was created either.

d@ubuntu:~/Documents/logstash-6.4.2/bin$ ./logstash
Sending Logstash logs to /home/d/Documents/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-10-04T12:32:10,040][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-10-04T12:32:10,099][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"29dce12b-a71f-415f-9e71-4e68702eacf4", :path=>"/home/d/Documents/logstash-6.4.2/data/uuid"}
[2018-10-04T12:32:10,809][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-04T12:32:13,625][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-04T12:32:14,152][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-10-04T12:32:14,166][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-10-04T12:32:14,445][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-10-04T12:32:14,565][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-10-04T12:32:14,568][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-10-04T12:32:14,701][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-10-04T12:32:14,762][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-10-04T12:32:14,794][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-10-04T12:32:15,104][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4a461065 run>"}
[2018-10-04T12:32:15,216][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-10-04T12:32:15,240][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2018-10-04T12:32:15,341][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["/home/d/pcap/test.csv"], id=>"5f0a1b5379a03722fad3e4ebbac4feeac072610f7d703b140b966224506698f1", sincedb_path=>"/var/log/tracking_no", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_26cd4a5b-035e-4695-a928-1ff7fef89209", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc">
  Error: Permission denied - /var/log/tracking_no
  Exception: Errno::EACCES
  Stack: org/jruby/RubyFile.java:366:in `initialize'
org/jruby/RubyIO.java:1154:in `open'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1167:in `block in touch'
org/jruby/RubyArray.java:1734:in `each'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1161:in `touch'
/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/sincedb_collection.rb:22:in `initialize'
/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:62:in `build_watch_and_dependencies'
/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:56:in `initialize'
/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:332:in `start_processing'
/home/d/Documents/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337:in `run'
/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
/home/d/Documents/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[2018-10-04T12:32:15,594][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-10-04T12:32:16,349][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2018-10-04T12:32:16,360][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["/home/d/pcap/test.csv"], id=>"5f0a1b5379a03722fad3e4ebbac4feeac072610f7d703b140b966224506698f1", sincedb_path=>"/var/log/tracking_no", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_26cd4a5b-035e-4695-a928-1ff7fef89209", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc">
  Error: Permission denied - /var/log/tracking_no
  Exception: Errno::EACCES
  Stack: org/jruby/RubyFile.java:366:in `initialize'
org/jruby/RubyIO.java:1154:in `open'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1167:in `block in touch'
org/jruby/RubyArray.java:1734:in `each'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1161:in `touch'
10

hey @bardie

changes you proposed helped push things forward.

what I additionally did is that I removed

sincedb_path => "/var/log/tracking_no"

line from the sample.config file and it worked

here this issue is described,

many thanks for your help!

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.