Not getting logs from filebeat at logstash

[Saurabh_Jambhule]

There are some sample apache logs in folder listed in paths, but still they not going to logstash
Here is my configuration,

filebeat:

filebeat:
  prospectors:
    -
      paths:
        - /home/saurabh/sample-log/*.log"
      input_type: log
  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    **hosts: ["localhost:5044"]**
  console:
    pretty: true

shipper:

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: mybeat
    rotateeverybytes: 10485760 # = 10MB

logstash:

input {
  beats {
    **port => 5044**
  }
  stdin { 
  }
}

filter{
  grok {
    match => ["message", "%{COMBINEDAPACHELOG}"]
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
  geoip {
    source => "clientip"
    target => "geoip"
    database => "/home/saurabh/Documents/GeoLiteCity.dat"
    add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
    add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
  }
  mutate {
    convert => ["[geoip][coordinates]", "float"]
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    manage_template => false
    **hosts => "localhost:9200"**
  }
}

My system config:
ubuntu_16.04
filebeat_1.2.3_amd64
elasticsearch-2.3.4
logstash_2.3.4-1_all
kibana_4.5.2_amd64

Thank you.

[andrewkroh]

Try these steps to collect some debug logging information from Filebeat:

1. Delete your Filebeat registry file to clear any state kept by Filebeat. Run rm /var/lib/filebeat/registry
2. Start Filebeat in the foreground and write debug to the console. Run filebeat -c path/to/your/filebeat.yml -e -v -d "*"

[Saurabh_Jambhule]

I got following output :

2016/07/27 13:09:42.777629 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/07/27 13:09:42.777713 logstash.go:106: INFO Max Retries set to: 3
2016/07/27 13:09:42.777894 outputs.go:126: INFO Activated logstash as output plugin.
2016/07/27 13:09:42.777914 outputs.go:126: INFO Activated console as output plugin.
2016/07/27 13:09:42.777962 publish.go:288: INFO Publisher name: saurabh
2016/07/27 13:09:42.792453 async.go:78: INFO Flush Interval set to: 1s
2016/07/27 13:09:42.792465 async.go:84: INFO Max Bulk Size set to: 1024
2016/07/27 13:09:42.792521 async.go:78: INFO Flush Interval set to: 1s
2016/07/27 13:09:42.792525 async.go:84: INFO Max Bulk Size set to: 2048
2016/07/27 13:09:42.792564 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.3
2016/07/27 13:09:42.792830 beat.go:173: INFO filebeat sucessfully setup. Start running.
2016/07/27 13:09:42.792855 registrar.go:68: INFO Registry file set to: /var/lib/filebeat/registry
2016/07/27 13:09:42.792867 prospector.go:133: INFO Set ignore_older duration to 0
2016/07/27 13:09:42.792871 prospector.go:133: INFO Set close_older duration to 1h0m0s
2016/07/27 13:09:42.792876 prospector.go:133: INFO Set scan_frequency duration to 10s
2016/07/27 13:09:42.792881 prospector.go:93: INFO Input type set to: log
2016/07/27 13:09:42.792886 prospector.go:133: INFO Set backoff duration to 1s
2016/07/27 13:09:42.792890 prospector.go:133: INFO Set max_backoff duration to 10s
2016/07/27 13:09:42.792901 prospector.go:113: INFO force_close_file is disabled
2016/07/27 13:09:42.792911 prospector.go:143: INFO Starting prospector of type: log
2016/07/27 13:09:42.792950 crawler.go:78: INFO All prospectors initialised with 0 states to persist
2016/07/27 13:09:42.792956 registrar.go:87: INFO Starting Registrar
2016/07/27 13:09:42.792966 publish.go:88: INFO Start sending events to output
2016/07/27 13:09:42.792979 spooler.go:77: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s

Still logs are not going into logstash.

[ruflin]

That seems to be only the setup phase. Can you share some more content?

[Saurabh_Jambhule]

Its working now. I mistakenly put double quotation at end of path
- /home/saurabh/sample-log/*.log"
Thank you!