Logstash csv upload not working

Elastic Stack Logstash
1

Using below config file
input {
file {
path => "C:\logstash\bin\saptest1.csv"
type => "core2"
start_position => "beginning"
}
}
filter {
csv {
columns => ["Material","Batch","Plant","SLoc"]
separator => ","
}
}
output {
stdout {
codec => rubydebug
}
}

CSV file has values for 4 columns separated by ,

Im getting below output and index not created

{:timestamp=>"2016-04-26T09:59:54.305000+0530", :message=>"Reading config file", :config_file=>"C:/logstash/bin/sap.conf", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
{:timestamp=>"2016-04-26T09:59:54.365000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-26T09:59:54.385000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-26T09:59:54.405000+0530", :message=>"config LogStash::Codecs::Plain/@charset = "UTF-8"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.405000+0530", :message=>"config LogStash::Inputs::File/@path = ["C:\\logstash\\bin\\saptest1.csv"]", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
"2016-04-26T10:00:23.927000+0530",

2

{:timestamp=>"2016-04-26T09:59:54.405000+0530", :message=>"config LogStash::Inputs::File/@type = "core2"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.405000+0530", :message=>"config LogStash::Inputs::File/@start_position = "beginning"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8">", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@add_field = {}", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@stat_interval = 1", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@discover_interval = 15", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@sincedb_write_interval = 15", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@delimiter = "\n"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.415000+0530", :message=>"config LogStash::Inputs::File/@ignore_older = 86400", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.425000+0530", :message=>"config LogStash::Inputs::File/@close_older = 3600", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}

3

{:timestamp=>"2016-04-26T09:59:54.425000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"filter", :name=>"csv", :path=>"logstash/filters/csv", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-26T09:59:54.475000+0530", :message=>"config LogStash::Filters::CSV/@columns = ["Material", "Batch", "Plant", "SLoc"]", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.475000+0530", :message=>"config LogStash::Filters::CSV/@separator = ","", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}

{:timestamp=>"2016-04-26T09:59:54.475000+0530", :message=>"config LogStash::Filters::CSV/@add_tag = []", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}

{:timestamp=>"2016-04-26T09:59:54.475000+0530", :message=>"config LogStash::Filters::CSV/@remove_tag = []", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}

{:timestamp=>"2016-04-26T09:59:54.475000+0530", :message=>"config LogStash::Filters::CSV/@add_field = {}", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.475000+0530", :message=>"config

LogStash::Filters::CSV/@remove_field = []", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"config

LogStash::Filters::CSV/@periodic_flush = false", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}

4

{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"config LogStash::Filters::CSV/@source = "message"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"config LogStash::Filters::CSV/@quote_char = "\""", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"config LogStash::Filters::CSV/@autogenerate_column_names = true", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"config LogStash::Filters::CSV/@skip_empty_columns = false", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"config LogStash::Filters::CSV/@convert = {}", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:54.485000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-26T09:59:54.515000+0530", :message=>"starting agent", :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"207", :method=>"execute"}
{:timestamp=>"2016-04-26T09:59:54.515000+0530", :message=>"starting pipeline", :id=>"main", :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"461", :method=>"start_pipeline"}
{:timestamp=>"2016-04-26T09:59:54.875000+0530", :message=>"Registering file input", :path=>["C:\logstash\bin\saptest1.csv"], :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb", :line=>"171", :method=>"register"}
{:timestamp=>"2016-04-26T09:59:54.875000+0530", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"C:\Users\akatti/.sincedb_3380cea5010b7c573bd60788eba91cf8", :path=>["C:\logstash\bin\saptest1.csv"], :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb", :line=>"216", :method=>"register"}
{:timestamp=>"2016-04-26T09:59:54.885000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"rubydebug", :path=>"logstash/codecs/rubydebug", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-26T09:59:54.885000+0530", :message=>"_globbed_files: C:\logstash\bin\saptest1.csv: glob is: []", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}
{:timestamp=>"2016-04-26T09:59:54.885000+0530", :message=>"_globbed_files: C:\logstash\bin\saptest1.csv: glob is: ["C:\\logstash\\bin\\saptest1.csv"] because glob did not work", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"349", :method=>"_globbed_files"}
{:timestamp=>"2016-04-26T09:59:54.885000+0530", :message=>"_discover_file: C:\logstash\bin\saptest1.csv: new: C:\logstash\bin\saptest1.csv (exclude is [])", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"310", :method=>"_discover_file"}

5

{:timestamp=>"2016-04-26T09:59:54.895000+0530", :message=>"_discover_file: C:\logstash\bin\saptest1.csv: skipping because it was last modified more than 86400.0 seconds ago", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"330", :method=>"_discover_file"}
{:timestamp=>"2016-04-26T09:59:54.895000+0530", :message=>"config LogStash::Codecs::RubyDebug/@metadata = false", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:55.225000+0530", :message=>"config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug metadata=>false>", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:55.225000+0530", :message=>"config LogStash::Outputs::Stdout/@workers = 1", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-26T09:59:55.235000+0530", :message=>"Will start workers for output", :worker_count=>1, :class=>LogStash::Outputs::Stdout, :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb", :line=>"77", :method=>"register"}
{:timestamp=>"2016-04-26T09:59:55.235000+0530", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>8, :batch_size=>125, :batch_delay=>5, :max_inflight=>1000, :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"188", :method=>"start_workers"}
{:timestamp=>"2016-04-26T09:59:55.235000+0530", :message=>"Pipeline main started", :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"465", :method=>"start_pipeline"}
{:timestamp=>"2016-04-26T10:00:00.235000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-04-26T10:00:05.235000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-04-26T10:00:08.903000+0530", :message=>"_globbed_files: C:\logstash\bin\saptest1.csv: glob is: []", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}
{:timestamp=>"2016-04-26T10:00:08.903000+0530", :message=>"_globbed_files: C:\logstash\bin\saptest1.csv: glob is: ["C:\\logstash\\bin\\saptest1.csv"] because glob did not work", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"349", :method=>"_globbed_files"}
{:timestamp=>"2016-04-26T10:00:10.243000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-04-26T10:00:15.243000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-04-26T10:00:20.253000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}

6

Hint: Post logs like this on pastebin or a similar service.

{:timestamp=>"2016-04-26T09:59:54.885000+0530", :message=>"globbedfiles: C:\logstash\bin\saptest1.csv: glob is: ", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"346", :method=>"globbedfiles"}

This indicates that the filename pattern (well, it's not really a pattern but it's treated as such) C:\logstash\bin\saptest1.csv doesn't match any files. This typically indicates a typo or that Logstash doesn't have permissions to access one of the directories, but in this particular case I suspect the problem is that you're using backslashes instead of forward slashes. At least previous versions of Logstash has had problems with backslashes on Windows.

{:timestamp=>"2016-04-26T09:59:54.895000+0530", :message=>"discoverfile: C:\logstash\bin\saptest1.csv: skipping because it was last modified more than 86400.0 seconds ago", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.1/lib/filewatch/watch.rb", :line=>"330", :method=>"discoverfile"}

Adjust the ignore_older option if you want to read files older than 24 hours.

1 Like
7

Thanks Magnus for that, it works

I am now trying to create index in ES with below conf

input {
file {
path => "C:/logstash/bin/saptest1.csv"
type => "core2"
start_position => "beginning"
ignore_older => 0
}
}
filter {
csv {
columns => ["Material","Batch","Plant","SLoc"]
separator => ","
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost"
index => "sap"
workers => 1
}
}

Its not creating index ..below is the log
http://pastebin.com/4A1vtsvM

8

OK was able to get that top work....My csv file has first row as column header. How can use the first row as column name ?

9

My csv file has first row as column header. How can use the first row as column name ?

That's currently not possible. See below.

10

Thanks