Hi guys. I have a server running beats and another server that has a docker container running logstash. Both are at aws. It seems that the connection is being made. From my debug.log from logstash I see this.
{:timestamp=>"2016-03-01T22:14:56.545000+0000", :message=>"Beats input: decoding this event with the codec", :target_field_value=>"{"sql":"SELECT \"public\".\"apiv3_authorizations\".* FROM \"public\".\"apiv3_authorizations\" WHERE \"public\".\"apiv3_authorizations\".\"project_id\" IS NULL AND \"public\".\"apiv3_authorizations\".\"user_id\" IS NULL LIMIT 1","name":"Apiv3::Authorization Load","connection_id":47308764521820,"statement_name":null,"binds":,"duration":0.6,"request_id":"437da433-eaf9-4653-9ddb-e6b5dab1da50","source":"unknown","tags":["request"],"@timestamp":"2016-03-01T22:14:49.395Z","@version":"1"}", :level=>:debug, :file=>"logstash/inputs/beats_support/connection_handler.rb", :line=>"55", :method=>"process"}
This is definitely from my beats server. Although the output never is generated. Nothing ends up in kibana and nothing gets output to a file on the logstash server. Any help would be great.
Here is my logstash config
input {
beats {
port => 5044
codec => json_lines
}
}
output {
file {
codec => plain
path => "/var/log/logstash/logstash.log"
}
elasticsearch {
codec => json_lines
hosts => ["awsURL:80"]
ssl => false
sniffing => false
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
and my beats config
filebeat:
# List of prospectors to fetch data.
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
paths:
#- /var/log/*.log
- /var/app/current/log/logstasher.log
encoding: plain
input_type: log
scan_frequency: 2s
registry_file: /var/lib/filebeat/registry
output:
### Logstash as output
logstash:
hosts: ["myip:5044"]
bulk_max_size: 1024
path: "/tmp/filebeat"
filename: filebeat
rotate_every_kb: 10000
pretty: true
logging:
to_files: true
# To enable logging to files, to_files option has to be set to true
files:
# The directory where the log files will written to.
path: /var/log/filebeat
# The name of the files where the logs are written to.
name: filebeat.log
# Configure log file size limit. If limit is reached, log file will be
# automatically rotated
rotateeverybytes: 10485760 # = 10MB
# Number of rotated log files to keep. Oldest files will be deleted first.
keepfiles: 5
# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are beat, publish, service
# Multiple selectors can be chained.
#selectors: [ ]
# Sets log level. The default log level is error.
# Available log levels are: critical, error, warning, info, debug
level: debug