Logstash conf file unable to read file for csv import

I am trying to get log stash to import my csv data load to elastic search cluster and view in Kibana. Logstash/ES/Kibana instances are all up and running but LS does not create the index.

Below is my Logstash.conf file

input {
file {
path => "/data/cars.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {
separator => ","
columns => ["maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type"]
}

mutate {convert => ["maker", "string"] }
mutate {convert => ["model", "string"] }
mutate {convert => ["mileage", "float"] }
mutate {convert => ["manufacture_year", "integer"] }
mutate {convert => ["engine_displacement", "integer"] }
mutate {convert => ["engine_power", "integer"] }
mutate {convert => ["body_type", "string"] }
mutate {convert => ["color_slug", "string"] }
mutate {convert => ["stk_year", "string"] }
mutate {convert => ["transmission", "string"] }
mutate {convert => ["door_count", "integer"] }
mutate {convert => ["seat_count", "integer"] }
mutate {convert => ["fuel_type", "string"] }
}

output {

elasticsearch { hosts => ["http://10.0.10.226:9200","http://10.0.10.227:9200","http://10.0.10.228:9200","http://10.0.10.229:9200"]
index => "cars"
document_type => "sold_cars"
}
stdout {codec => rubydebug }
}

Below is the output in the log file

[2017-11-21T00:19:56,669][INFO ][logstash.pipeline ] Pipeline main started
[2017-11-21T00:19:56,872][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-11-21T00:29:30,503][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-11-21T00:29:30,515][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-11-21T00:29:45,625][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-21T00:29:45,628][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-11-21T00:29:47,083][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.10.226:9200/, http://10.0.10.227:9200/, http://10.0.10.228:9200/, http://10.0.10.229:9200/]}}
[2017-11-21T00:29:47,084][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.10.226:9200/, :path=>"/"}
[2017-11-21T00:29:47,170][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.0.10.226:9200/"}
[2017-11-21T00:29:47,208][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.10.227:9200/, :path=>"/"}
[2017-11-21T00:29:47,217][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.0.10.227:9200/"}
[2017-11-21T00:29:47,225][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.10.228:9200/, :path=>"/"}
[2017-11-21T00:29:47,237][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.0.10.228:9200/"}
[2017-11-21T00:29:47,243][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.10.229:9200/, :path=>"/"}
[2017-11-21T00:29:47,251][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.0.10.229:9200/"}
[2017-11-21T00:29:47,259][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-11-21T00:29:47,264][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-11-21T00:29:47,288][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.10.226:9200", "http://10.0.10.227:9200", "http://10.0.10.228:9200", "http://10.0.10.229:9200"]}
[2017-11-21T00:29:47,293][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-11-21T00:29:47,509][INFO ][logstash.pipeline ] Pipeline main started
[2017-11-21T00:29:47,546][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

The file is not read and the system just hangs for an indefinite time.

Does the user that Logstash runs as have access to the CSV file? Try bumping up the log level and look for "discover" and "glob" in the Logstash log.

Tried the below

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logs tash.conf --debug

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2017-11-23 05:22:40.742 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

I am not sure if there is another instance running. Unable to locate .lock file in /var/lib folder to delete.

Kindly advise

I don't the above issue by since I stopped the other default instance running. But when I look into the log I get the below

[2017-11-23T06:24:40,461][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-11-23T06:24:40,469][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-11-23T06:24:40,480][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.10.226:9200", "http://10.0.10.227:9200", "http://10.0.10.228:9200", "http://10.0.10.229:9200"]}
[2017-11-23T06:24:40,482][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-11-23T06:24:40,734][INFO ][logstash.pipeline ] Pipeline main started
[2017-11-23T06:24:40,817][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-11-23T06:24:58,416][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-11-23T06:24:58,429][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

I have tried to fix all the above issue. I am now getting the below error

[ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"}

I have tried to use hexdump and was unable to find any issues in the file

Below is the config file.

input {
file {
path => "/home/data/cars.csv"
start_position => "beginning"
sincedb_path => "/dev/null"

}

}

filter {
csv {
separator => ","
columns => ["mpg", "cylinders", "cubicinches", "hp", "weightlbs", "time-to-60", "year", "brand"]

}
mutate {convert => ["mpg", "float"] }
mutate {convert => ["cylinders", "integer"] }
mutate {convert => ["cubicinches", "integer"] }
mutate {convert => ["hp", "integer"] }
mutate {convert => ["weightlbs", "integer"] }
mutate {convert => ["time-to-60", "integer"] }
mutate {convert => ["year", "integer"] }
mutate {convert => ["brand", "string"] }

}

output {

elasticsearch { hosts => ["http://10.0.10.226:9200","http://10.0.10.227:9200","http://10.0.10.228:9200","http://10.0.10.229:9200"]
index => "cars"
document_type => "cars_brand"
}
stdout {codec => rubydebug }

The error message indicates that you have a file in your config directory that shouldn't be there. The directory passed to Logstash with the -f option must only contain pipeline configuration files (with inputs, outputs, and filters). Not logstash.yml, jvm.options, or any other configuration file.

I am using logstash 5.6.4. I have the config file in /etc/logstash/conf.d folder and don't have any other files. I have logstash.yml under /etc/logstash
When I check my log files it is continuously loading and does not end. I dont see any other error now

[2017-11-27T03:59:48,418][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-27T03:59:48,421][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}

I was able to fix the issue. It appeared to a execute permission issue of the input file.

Thanks for your help

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.