Loading csv using logstash into Elastic

Elastic Stack Logstash
1

Hi,

I am using Logstash version 6.4.1 and trying to load a csv file into Elastic via Logstash. I am unable to load the file and keep getting the following error.

[2018-09-23T09:01:16,937][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::File start_position=>"beginning", path=>["C:\\softwares\\logstash-6.4.1\\data\\test.csv"], id=>"e315cf4128a86616a22c5b516ffdf15ad0472ec28c819380c5b38f0c22520e5c", sincedb_path=>"C:\\softwares\\logstash-6.4.1\\data", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b50149be-87ae-44a8-aac9-068ea0c5e0a5", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc">", :error=>"The "sincedb_path" argument must point to a file, received a directory: "C:\softwares\logstash-6.4.1\data"", :thread=>"#<Thread:0x340a87 run>"}
[2018-09-23T09:01:17,462][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<ArgumentError: The "sincedb_path" argument must point to a file, received a directory: "C:\softwares\logstash-6.4.1\data">, :backtrace=>["C:/softwares/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:280:in register'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:242:inregister_plugin'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:253:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:253:in register_plugins'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:396:instart_inputs'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:294:in start_workers'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:200:inrun'", "C:/softwares/logstash-6.4.1/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0x340a87 run>"}
[2018-09-23T09:01:17,917][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

The csv is as below
col1,col2
1,2
3,4

The config file looks like as below
input {
file {
path => "C:\softwares\logstash-6.4.1\data\test.csv"
start_position => "beginning"
sincedb_path => "C:\softwares\logstash-6.4.1\data"
}
}

filter {
csv {
skip_header => true
separator => ","
autodetect_column_names => true
}
}

output {
elasticsearch {
hosts => "http://DCInnive:9200"
index => "clay_test"
}
}

Kindly request for help.

Thank you.

2

I think this error message is quite clear and explains what is wrong.

3

Hi Christian,

Thank you for the help. I tried pointing to a blank file. The error stopped but I never got a success message. The cmd never stopped. I waited a while and then killed the process.

I am sorry, if it is a basic question. Little new to the area and I didn't find how exactly I could resolve the issue.

4

If you are new to Logstash I would recommend going through this blog post, which provides an introduction to how to work with Logstash.

5

Thank you. That would surely help. I was trying to do the sample file load as given in the blog but I never got the output. It always happens when I try loading a file.
Below is where it gets struck.
logstash -f logstash-test.config
Sending Logstash logs to C:/softwares/logstash-6.4.1/logs which is now configured via log4j2.properties
[2018-09-25T10:50:02,958][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-25T10:50:03,280][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.1"}
[2018-09-25T10:50:04,909][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-09-25T10:50:05,368][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x614af9 sleep>"}
[2018-09-25T10:50:05,408][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2018-09-25T10:50:05,408][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-09-25T10:50:05,650][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

There is basically no response after this. Below is the config file I used.

input {
file {
path => "C:\softwares\logstash-6.4.1\data\sample.log"
start_position => "beginning"
sincedb_path => "nul"
}
}

filter {
}

output {
stdout{
codec => rubydebug
}
}

The file "sample.log" had a welcome message.

Kindly request your help.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.