Logstash not sending data to elasticsearch when run as daemon

Elastic Stack Logstash
1

Hi!

Running into an issue where if I run ./logstash -f manually everything is fine. Reads a log file, logs show up in ES.

When starting the service, however, it starts normally and no errors are seen, but no data is populated in ES.

from logstash.log after starting daemon via 'service logstash start':

{:timestamp=>"2016-06-08T14:43:16.846000-0400", :message=>"starting agent", :level=>:info}
{:timestamp=>"2016-06-08T14:43:16.850000-0400", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.005000-0400", :message=>"Registering file input", :path=>["/var/log/icinga2/icinga2.log"], :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.011000-0400", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/var/lib/logstash/.sincedb_b4836ea57d9b1654a4a5f18974e89332", :path=>["/var/log/icinga2/icinga2.log"], :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.262000-0400", :message=>"Using mapping template from", :path=>nil, :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.553000-0400", :message=>"Attempting to install template", :manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.933000-0400", :message=>"New Elasticsearch output", :class=>"LogStash::Outputs::ElasticSearch", :hosts=>["10.15.254.20:9200"], :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.936000-0400", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>4, :batch_size=>125, :batch_delay=>5, :max_inflight=>500, :level=>:info}
{:timestamp=>"2016-06-08T14:43:17.955000-0400", :message=>"Pipeline main started"}'

It looks like it found both the configuration and input file and didn't complain about permissions or anything, started the pipeline...but I get nothing. Generally it seems this problem is related to user permissions, but since no permissions errors were seen in the log I do not believe that to be the case.

Unfortunately the logs don't provide much to go on, has anyone else experienced an issue similar before?

Thanks,

2

Can the logstash user read /var/log/icinga2/icinga2.log and read/write to /var/lib/logstash/? /var/lib/logstash/ is where the file position is kept in the .sincedb* file(s). Also until any new messages are appended to the file no new data will be sent. You may want to try using beginning for start_position:

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position

3

Thanks for the reply Mike.

I guess I spoke too soon about having my file permissions correct. I set the directory permissions to '-R go+rX' and that was the problem. I had assumed the logfile would have yelled at me if it couldn't read the input file but I guess not.