I need help with my logstash config

Elastic Stack Logstash
1

Hello,

Could someone help me with the message below. Im not sure if the issue is my configuration or the .csv file.

This is a snippet from my debug

Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"453", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"453", :method=>"flush"}
_globbed_files: /opt/logstash/csv/.csv: glob is: ["/opt/logstash/csv/alohamon-alerts-2016-10-07.csv"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"453", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"453", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"453", :method=>"flush"}
_globbed_files: /opt/logstash/csv/.csv: glob is: ["/opt/logstash/csv/alohamon-alerts-2016-10-07.csv"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}

My logstash.conf

input {
file {
path => "/opt/logstash/csv/*.csv"
type => "csv" # a type to identify those logs (will need this later)
start_position => "beginning"
}
}

filter {
csv {
columns => ["ID","URL","Brouha","Action","Last Action before Clear","Resolve/Close Reason","In Process","Chronic","Service Affecting","Created","AM PM","From","Till","Duration (minutes)","Customers","STBs","TTA","TTI","TTS","TTR","By","Region","DAC","Division","Device","IP","Upstreams","Reason","Comment","Root Cause","Corrective Action Taken","SI Ticket","JB Ticket"]

separator => ","

}
mutate {
convert => ["TempOut", "float"]
}
}

#output {
#elasticsearch {

hosts => "localhost" # it used to be "host" and "port" pre-2.0

index => "avsdata"

#}

stdout {

codec => rubydebug

}

#}

output {
stdout { codec => rubydebug }
}

Thank you,

Keith

2

What help do you want? It's a debug message, you can ignore it.

3

Hi Mark,

The issue is that the fields column names from my CSV are not populating in my index when I add it to kibana. When I curl the indices its not showing a docs count. logstash starts and I tested the config. Im not sure how to trouble shoot this or where to look.

[root@localhost ~]# curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 1 1 1 0 6.9kb 6.9kb
yellow open avsdata 3 2 0 0 477b 477b
[root@localhost ~]#

The way I understand this is that the CSV file should be injested by logstash, filtered, and sent to elastic search. It writes to an index/indices? I should be able to add the index to Kibana that I created with curl xput. The CSV column names should populate under indices/fields in Kibana. I should be able to select what fields I want to pull data from when using terms to create a pie chart. When I try to discover the data in Kibana I receive No results found. I date set year to date.

curl -XPUT 'http://localhost:9200/avsdata/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2

    }
}

}'

Thank you,

Keith

4

So when you start LS does anything show in stdout?

5

Hey Mark,

This is the output. I added -verbose.

[keith@localhost bin]$ ./logstash -f logstash.conf --verbose
Settings: Default pipeline workers: 1
Registering file input {:path=>["/opt/logstash/bin/csv/.csv"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/home/keith/.sincedb_16a6cbe744da9d5cf03a712932ae498c", :path=>["/opt/logstash/bin/csv/.csv"], :level=>:info}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost"], :level=>:info}
Starting pipeline {:id=>"base", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_inflight=>125, :level=>:info}
Pipeline started {:level=>:info}

6

Things to look into:

  • Logstash could be tailing the file. Check the sincedb file (see logs for its path) to be sure. Delete it or set sincedb_path to /dev/null.
  • If the input file is older than 24 hours you need to adjust the ignore_older option.

Until you have gotten Logstash to read your CSV file and output to your stdout output do not waste time by looking in Elasticsearch and Kibana.

7

Where is the ignore_older? I dont have it listed in my logstash config.

Thanks,

Keith

8

Please see the file input plugin's documention.

9

Hi Mark,

I was able to generate an output. It appears that only part of the csv is read and logstash hangs. It does not appear to be writing to the index. Ive tried restarting logstash a few times and it stops at the same location in the file both times. I uploaded a new csv file this is how I was able to get logstash to generate stdout. I added ignore_older => 0 to the input part of the config.

Thanks,

Keith