Logstash not showing any data when input is configured with filebeat

[Vijay_Chander]

filebeat.yml file

filebeat.prospectors:

• type: log
  paths:
  • "/home/ec2-user/logs/sas_logs/*.log"
    output.logstash:
    hosts: ["localhost:5044"]

logstash.conf file

input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time_stamp}\s+%{LOGLEVEL:loglevel}\s+[%{NUMBER:num}]\s+:%{USERNAME:user}\s+-\s+%{GREEDYDATA:msg}" }
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "a3_logs"
}
stdout {
codec => rubydebug
}
}

log format: 2018-07-09T16:20:30,799 INFO [00000006] :sasinst - NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.

When I run logstash, it is stuck at Pipeline Started. I'm not able to understand the issue.

Can anyone please help me ??

[pierhugues (Pier-Hugues Pellerin)]

Hello @Vijay_Chander, by looking at your filebeat configuration, I believe you have an indentation issue in your configuration and you also need to enable the logstash output.

filebeat.prospectors:
  - type: log    
    paths:
      - "/home/ec2-user/logs/sas_logs/*.log"
output.logstash:
  enabled: true
  hosts: ["localhost:5044"]

Also starting filebeat in debug mode should give us a a better idea is there is others issues, you can do that by running the following command:

./filebeat -v -e -d "*" -c ./myconfig.yml

[Vijay_Chander]

Hi @pierhugues,

I've tried as you suggested but still no luck. Below is my filebeat log

2018-08-09T05:32:11.870Z DEBUG [input] log/input.go:175 input states cleaned up. Before: 5, After: 5, Pending: 0

logstash log

[2018-08-09T05:27:28,029][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-09T05:27:28,141][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1e67cb22 run>"}
[2018-08-09T05:27:28,187][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-09T05:27:28,322][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-08-09T05:27:28,614][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Can you please suggest any other way??

[pierhugues (Pier-Hugues Pellerin)]

@Vijay_Chander lets concentrate on the Filebeat side, If you want filebeat with the options I've said in my previous comment the log should be a lot bigger. Can you include a more complete log?