Logstash file input

Trying to figure out an issue with file input

It just doesn't seem to be reading the file. It is a suricata eve.json file

  • permissions look fine (world readable)
  • verbose output says it registers the file
  • no output to elasticsearch
  • no output to standard out either

:timestamp=>"2016-08-15T17:15:16.313000-0700", :message=>"starting agent", :level=>:info}
{:timestamp=>"2016-08-15T17:15:16.318000-0700", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2016-08-15T17:15:16.663000-0700", :message=>"Registering file input", :path=>["/var/log/suricata/eve.json"], :level=>:info}
{:timestamp=>"2016-08-15T17:15:16.776000-0700", :message=>"** WARNING ** Detected UNSAFE options in elasticsearch output configuration!\n** WARNING ** You have enabled encryption but DISABLED certificate verification.\n** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true", :level=>:warn}
{:timestamp=>"2016-08-15T17:15:17.239000-0700", :message=>"Using mapping template from", :path=>nil, :level=>:info}
{:timestamp=>"2016-08-15T17:15:17.453000-0700", :message=>"Attempting to install template", :manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info}
{:timestamp=>"2016-08-15T17:15:17.832000-0700", :message=>"New Elasticsearch output", :class=>"LogStash::Outputs::ElasticSearch", :hosts=>[""], :level=>:info}
{:timestamp=>"2016-08-15T17:15:17.943000-0700", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
{:timestamp=>"2016-08-15T17:15:17.983000-0700", :message=>"Pipeline main started"}

@elk-receiver ~]# ls -l /var/log/suricata/eve.json
-rw-r--r--. 1 suricata suricata 498738317 Aug 15 17:18 /var/log/suricata/eve.json

cat /etc/logstash/conf.d/10-ids-suricata-input.conf

input {
file {
path => ["/var/log/suricata/eve.json"]
sincedb_path => ["/var/lib/logstash/suricata.sincedb"]
start_position => "beginning"
codec => "json"
type => "Suricata"
tags => ["Suricata","IDS"]

]# cat /etc/logstash/conf.d/50-ids-suricata-output.conf
output {
if [type] == "Suricata" {
elasticsearch {
hosts => [""]
ssl => true
ssl_certificate_verification => false
keystore => "/var/certs/elkstack.jks"
keystore_password => password
user => "user"
password => password
index => "unv_suricata-%{+YYYY.MM.dd}"
stdout {
codec => rubydebug

I have other TCP/UDP inputs and the elasticsearch output working but have it striped down to this at the moment trying to find the issue.

any ideas would be appreciated

Have you tried deleting the sincedb file?

I fiddled with that a bit, that is a new path in the configuration file and
I didn't specify the file, just directory.

There was an error in the log saying it needed a file specified. I did that
and it never created a file so I touched it and the size changed but still
never worked.

I can try deleting that file again and see what happens I guess

deleted the file, restarted logstash

  • regenerated traffic to update the eve.json file
  • no change, no data loaded in to elasticsearch nor stdout
  • the sincedb file wasn't created either
  • running as the logstash user
  • logstash user has permissions

root 22226 0.0 0.0 193436 2788 pts/0 S+ 08:00 0:00 sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --log /var/log/logstash/logstash.log --verbose

logstash 22227 32.6 5.4 3633364 219192 pts/0 Sl+ 08:00 0:21 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d --log /var/log/logstash/logstash.log --verbose

[@elk-receiver ~]# ls -la /var/lib/logstash/
total 4
drwxrwxr-x. 2 logstash logstash 6 Aug 16 07:59 .