# I need help with my logstash config **URL:** \ **Category:** Logstash\ **Created:** [October 7, 2016, 5:32pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516 "2016-10-07T17:32:48Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![keyboard112](https://avatars.discourse-cdn.com/v4/letter/k/41988e/32.png) [@keyboard112](https://discuss.elastic.co/u/keyboard112)\ **Post date:** [October 7, 2016, 5:32pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/1 "2016-10-07T17:32:48Z")
Hello, Could someone help me with the message below. Im not sure if the issue is my configuration or the .csv file. This is a snippet from my debug Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"453", :method=\>"flush"} Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"453", :method=\>"flush"} \_globbed\_files: /opt/logstash/csv/_.csv: glob is: ["/opt/logstash/csv/alohamon-alerts-2016-10-07.csv"] {:level=\>:debug, :file=\>"filewatch/watch.rb", :line=\>"346", :method=\>"\_globbed\_files"} Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"453", :method=\>"flush"} Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"453", :method=\>"flush"} Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"453", :method=\>"flush"} \_globbed\_files: /opt/logstash/csv/_.csv: glob is: ["/opt/logstash/csv/alohamon-alerts-2016-10-07.csv"] {:level=\>:debug, :file=\>"filewatch/watch.rb", :line=\>"346", :method=\>"\_globbed\_files"} My logstash.conf input { file { path =\> "/opt/logstash/csv/\*.csv" type =\> "csv" # a type to identify those logs (will need this later) start\_position =\> "beginning" } } filter { csv { columns =\> ["ID","URL","Brouha","Action","Last Action before Clear","Resolve/Close Reason","In Process","Chronic","Service Affecting","Created","AM PM","From","Till","Duration (minutes)","Customers","STBs","TTA","TTI","TTS","TTR","By","Region","DAC","Division","Device","IP","Upstreams","Reason","Comment","Root Cause","Corrective Action Taken","SI Ticket","JB Ticket"] ``` separator => "," ``` } mutate { convert =\> ["TempOut", "float"] } } #output { #elasticsearch { # hosts =\> "localhost" # it used to be "host" and "port" pre-2.0 # index =\> "avsdata" #} # stdout { # codec =\> rubydebug # } #} output { stdout { codec =\> rubydebug } } Thank you, Keith ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 8, 2016, 5:51am UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/2 "2016-10-08T05:51:23Z")
What help do you want? It's a debug message, you can ignore it. ---
**Author:** ![keyboard112](https://avatars.discourse-cdn.com/v4/letter/k/41988e/32.png) [@keyboard112](https://discuss.elastic.co/u/keyboard112)\ **Post date:** [October 8, 2016, 12:58pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/3 "2016-10-08T12:58:18Z")
Hi Mark, The issue is that the fields column names from my CSV are not populating in my index when I add it to kibana. When I curl the indices its not showing a docs count. logstash starts and I tested the config. Im not sure how to trouble shoot this or where to look. [root@localhost ~]# curl 'localhost:9200/\_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open .kibana 1 1 1 0 6.9kb 6.9kb yellow open avsdata 3 2 0 0 477b 477b [root@localhost ~]# The way I understand this is that the CSV file should be injested by logstash, filtered, and sent to elastic search. It writes to an index/indices? I should be able to add the index to Kibana that I created with curl xput. The CSV column names should populate under indices/fields in Kibana. I should be able to select what fields I want to pull data from when using terms to create a pie chart. When I try to discover the data in Kibana I receive No results found. I date set year to date. curl -XPUT '[http://localhost:9200/avsdata/](http://localhost:9200/avsdata/)' -d '{ "settings" : { "index" : { "number\_of\_shards" : 3, "number\_of\_replicas" : 2 ``` } } ``` }' Thank you, Keith ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 8, 2016, 8:36pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/4 "2016-10-08T20:36:21Z")
So when you start LS does anything show in stdout? ---
**Author:** ![keyboard112](https://avatars.discourse-cdn.com/v4/letter/k/41988e/32.png) [@keyboard112](https://discuss.elastic.co/u/keyboard112)\ **Post date:** [October 9, 2016, 1:30pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/5 "2016-10-09T13:30:13Z")
Hey Mark, This is the output. I added -verbose. [keith@localhost bin]$ ./logstash -f logstash.conf --verbose Settings: Default pipeline workers: 1 Registering file input {:path=\>["/opt/logstash/bin/csv/_.csv"], :level=\>:info} No sincedb\_path set, generating one based on the file path {:sincedb\_path=\>"/home/keith/.sincedb\_16a6cbe744da9d5cf03a712932ae498c", :path=\>["/opt/logstash/bin/csv/_.csv"], :level=\>:info} Using mapping template from {:path=\>nil, :level=\>:info} Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "omit\_norms"=\>true}, "dynamic\_templates"=\>[{"message\_field"=\>{"match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}, "fields"=\>{"raw"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true, "ignore\_above"=\>256}}}}}, {"float\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"float", "mapping"=\>{"type"=\>"float", "doc\_values"=\>true}}}, {"double\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"double", "mapping"=\>{"type"=\>"double", "doc\_values"=\>true}}}, {"byte\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"byte", "mapping"=\>{"type"=\>"byte", "doc\_values"=\>true}}}, {"short\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"short", "mapping"=\>{"type"=\>"short", "doc\_values"=\>true}}}, {"integer\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"integer", "mapping"=\>{"type"=\>"integer", "doc\_values"=\>true}}}, {"long\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"long", "mapping"=\>{"type"=\>"long", "doc\_values"=\>true}}}, {"date\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"date", "mapping"=\>{"type"=\>"date", "doc\_values"=\>true}}}, {"geo\_point\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"geo\_point", "mapping"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "doc\_values"=\>true}, "@version"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true}, "geoip"=\>{"type"=\>"object", "dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip", "doc\_values"=\>true}, "location"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}, "latitude"=\>{"type"=\>"float", "doc\_values"=\>true}, "longitude"=\>{"type"=\>"float", "doc\_values"=\>true}}}}}}}, :level=\>:info} New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["localhost"], :level=\>:info} Starting pipeline {:id=\>"base", :pipeline\_workers=\>1, :batch\_size=\>125, :batch\_delay=\>5, :max\_inflight=\>125, :level=\>:info} Pipeline started {:level=\>:info} ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 9, 2016, 2:29pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/6 "2016-10-09T14:29:47Z")
Things to look into: - Logstash could be tailing the file. Check the sincedb file (see logs for its path) to be sure. Delete it or set `sincedb_path` to /dev/null. - If the input file is older than 24 hours you need to adjust the `ignore_older` option. Until you have gotten Logstash to read your CSV file and output to your stdout output do not waste time by looking in Elasticsearch and Kibana. ---
**Author:** ![keyboard112](https://avatars.discourse-cdn.com/v4/letter/k/41988e/32.png) [@keyboard112](https://discuss.elastic.co/u/keyboard112)\ **Post date:** [October 9, 2016, 7:45pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/7 "2016-10-09T19:45:33Z")
Where is the ignore\_older? I dont have it listed in my logstash config. Thanks, Keith ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 9, 2016, 8:05pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/8 "2016-10-09T20:05:14Z")
Please see the file input plugin's documention. ---
**Author:** ![keyboard112](https://avatars.discourse-cdn.com/v4/letter/k/41988e/32.png) [@keyboard112](https://discuss.elastic.co/u/keyboard112)\ **Post date:** [October 10, 2016, 7:12pm UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/9 "2016-10-10T19:12:07Z")
Hi Mark, I was able to generate an output. It appears that only part of the csv is read and logstash hangs. It does not appear to be writing to the index. Ive tried restarting logstash a few times and it stops at the same location in the file both times. I uploaded a new csv file this is how I was able to get logstash to generate stdout. I added ignore\_older =\> 0 to the input part of the config. Thanks, Keith ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:34am UTC](https://discuss.elastic.co/t/i-need-help-with-my-logstash-config/62516/10 "2017-07-06T04:34:52Z")