# Logstash conf file unable to read file for csv import **URL:** \ **Category:** Logstash\ **Created:** [November 21, 2017, 6:05am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514 "2017-11-21T06:05:02Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![shivakv](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@shivakv](https://discuss.elastic.co/u/shivakv)\ **Post date:** [November 21, 2017, 6:05am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/1 "2017-11-21T06:05:03Z")
I am trying to get log stash to import my csv data load to elastic search cluster and view in Kibana. Logstash/ES/Kibana instances are all up and running but LS does not create the index. Below is my Logstash.conf file input { file { path =\> "/data/cars.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," columns =\> ["maker", "model", "mileage", "manufacture\_year", "engine\_displacement", "engine\_power", "body\_type", "color\_slug", "stk\_year", "transmission", "door\_count", "seat\_count", "fuel\_type"] } mutate {convert =\> ["maker", "string"] } mutate {convert =\> ["model", "string"] } mutate {convert =\> ["mileage", "float"] } mutate {convert =\> ["manufacture\_year", "integer"] } mutate {convert =\> ["engine\_displacement", "integer"] } mutate {convert =\> ["engine\_power", "integer"] } mutate {convert =\> ["body\_type", "string"] } mutate {convert =\> ["color\_slug", "string"] } mutate {convert =\> ["stk\_year", "string"] } mutate {convert =\> ["transmission", "string"] } mutate {convert =\> ["door\_count", "integer"] } mutate {convert =\> ["seat\_count", "integer"] } mutate {convert =\> ["fuel\_type", "string"] } } output { elasticsearch { hosts =\> ["[http://10.0.10.226:9200](http://10.0.10.226:9200)","[http://10.0.10.227:9200](http://10.0.10.227:9200)","[http://10.0.10.228:9200](http://10.0.10.228:9200)","[http://10.0.10.229:9200](http://10.0.10.229:9200)"] index =\> "cars" document\_type =\> "sold\_cars" } stdout {codec =\> rubydebug } } Below is the output in the log file [2017-11-21T00:19:56,669][INFO][logstash.pipeline] Pipeline main started [2017-11-21T00:19:56,872][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2017-11-21T00:29:30,503][WARN][logstash.runner] SIGTERM received. Shutting down the agent. [2017-11-21T00:29:30,515][WARN][logstash.agent] stopping pipeline {:id=\>"main"} [2017-11-21T00:29:45,625][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [2017-11-21T00:29:45,628][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} [2017-11-21T00:29:47,083][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://10.0.10.226:9200/](http://10.0.10.226:9200/), [http://10.0.10.227:9200/](http://10.0.10.227:9200/), [http://10.0.10.228:9200/](http://10.0.10.228:9200/), [http://10.0.10.229:9200/](http://10.0.10.229:9200/)]}} [2017-11-21T00:29:47,084][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://10.0.10.226:9200/](http://10.0.10.226:9200/), :path=\>"/"} [2017-11-21T00:29:47,170][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.0.10.226:9200/](http://10.0.10.226:9200/)"} [2017-11-21T00:29:47,208][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://10.0.10.227:9200/](http://10.0.10.227:9200/), :path=\>"/"} [2017-11-21T00:29:47,217][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.0.10.227:9200/](http://10.0.10.227:9200/)"} [2017-11-21T00:29:47,225][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://10.0.10.228:9200/](http://10.0.10.228:9200/), :path=\>"/"} [2017-11-21T00:29:47,237][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.0.10.228:9200/](http://10.0.10.228:9200/)"} [2017-11-21T00:29:47,243][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://10.0.10.229:9200/](http://10.0.10.229:9200/), :path=\>"/"} [2017-11-21T00:29:47,251][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.0.10.229:9200/](http://10.0.10.229:9200/)"} [2017-11-21T00:29:47,259][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} [2017-11-21T00:29:47,264][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2017-11-21T00:29:47,288][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[http://10.0.10.226:9200](http://10.0.10.226:9200)", "[http://10.0.10.227:9200](http://10.0.10.227:9200)", "[http://10.0.10.228:9200](http://10.0.10.228:9200)", "[http://10.0.10.229:9200](http://10.0.10.229:9200)"]} [2017-11-21T00:29:47,293][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>250} [2017-11-21T00:29:47,509][INFO][logstash.pipeline] Pipeline main started [2017-11-21T00:29:47,546][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} The file is not read and the system just hangs for an indefinite time. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 21, 2017, 6:27am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/2 "2017-11-21T06:27:56Z")
Does the user that Logstash runs as have access to the CSV file? Try bumping up the log level and look for "discover" and "glob" in the Logstash log. ---
**Author:** ![shivakv](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@shivakv](https://discuss.elastic.co/u/shivakv)\ **Post date:** [November 23, 2017, 10:26am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/3 "2017-11-23T10:26:57Z")
Tried the below /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logs tash.conf --debug WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [FATAL] 2017-11-23 05:22:40.742 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting. I am not sure if there is another instance running. Unable to locate .lock file in /var/lib folder to delete. Kindly advise ---
**Author:** ![shivakv](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@shivakv](https://discuss.elastic.co/u/shivakv)\ **Post date:** [November 23, 2017, 11:51am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/4 "2017-11-23T11:51:21Z")
I don't the above issue by since I stopped the other default instance running. But when I look into the log I get the below [2017-11-23T06:24:40,461][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} [2017-11-23T06:24:40,469][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2017-11-23T06:24:40,480][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[http://10.0.10.226:9200](http://10.0.10.226:9200)", "[http://10.0.10.227:9200](http://10.0.10.227:9200)", "[http://10.0.10.228:9200](http://10.0.10.228:9200)", "[http://10.0.10.229:9200](http://10.0.10.229:9200)"]} [2017-11-23T06:24:40,482][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>250} [2017-11-23T06:24:40,734][INFO][logstash.pipeline] Pipeline main started [2017-11-23T06:24:40,817][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2017-11-23T06:24:58,416][WARN][logstash.runner] SIGTERM received. Shutting down the agent. [2017-11-23T06:24:58,429][WARN][logstash.agent] stopping pipeline {:id=\>"main"} ---
**Author:** ![shivakv](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@shivakv](https://discuss.elastic.co/u/shivakv)\ **Post date:** [November 26, 2017, 11:02am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/5 "2017-11-26T11:02:35Z")
I have tried to fix all the above issue. I am now getting the below error [ERROR][logstash.agent] Cannot create pipeline {:reason=\>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"} I have tried to use hexdump and was unable to find any issues in the file Below is the config file. input { file { path =\> "/home/data/cars.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" ``` } ``` } filter { csv { separator =\> "," columns =\> ["mpg", "cylinders", "cubicinches", "hp", "weightlbs", "time-to-60", "year", "brand"] } mutate {convert =\> ["mpg", "float"] } mutate {convert =\> ["cylinders", "integer"] } mutate {convert =\> ["cubicinches", "integer"] } mutate {convert =\> ["hp", "integer"] } mutate {convert =\> ["weightlbs", "integer"] } mutate {convert =\> ["time-to-60", "integer"] } mutate {convert =\> ["year", "integer"] } mutate {convert =\> ["brand", "string"] } } output { elasticsearch { hosts =\> ["[http://10.0.10.226:9200](http://10.0.10.226:9200)","[http://10.0.10.227:9200](http://10.0.10.227:9200)","[http://10.0.10.228:9200](http://10.0.10.228:9200)","[http://10.0.10.229:9200](http://10.0.10.229:9200)"] index =\> "cars" document\_type =\> "cars\_brand" } stdout {codec =\> rubydebug } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 26, 2017, 7:56pm UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/6 "2017-11-26T19:56:53Z")
The error message indicates that you have a file in your config directory that shouldn't be there. The directory passed to Logstash with the `-f` option must only contain pipeline configuration files (with inputs, outputs, and filters). Not logstash.yml, jvm.options, or any other configuration file. ---
**Author:** ![shivakv](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@shivakv](https://discuss.elastic.co/u/shivakv)\ **Post date:** [November 27, 2017, 9:02am UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/7 "2017-11-27T09:02:24Z")
I am using logstash 5.6.4. I have the config file in /etc/logstash/conf.d folder and don't have any other files. I have logstash.yml under /etc/logstash When I check my log files it is continuously loading and does not end. I dont see any other error now [2017-11-27T03:59:48,418][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [2017-11-27T03:59:48,421][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} ---
**Author:** ![shivakv](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@shivakv](https://discuss.elastic.co/u/shivakv)\ **Post date:** [November 30, 2017, 3:09pm UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/8 "2017-11-30T15:09:23Z")
I was able to fix the issue. It appeared to a execute permission issue of the input file. Thanks for your help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 28, 2017, 3:09pm UTC](https://discuss.elastic.co/t/logstash-conf-file-unable-to-read-file-for-csv-import/108514/9 "2017-12-28T15:09:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.