I have an issue diffucult to troubleshoot.
I had set up an architecture with filebeat as forwarder on my web APIs, a logstash listening to filebeat on port 5044 on a dedicated machine, and indexing log messages on a cluster of elasticsearh.
I then dicide to update the architecture like this : filebeat as forwarder on my web APIs, a redis broker (managed in aws), a logstash indexer on a dedicated machine, and the cluster of elasticsearh to store indices.
In the logstash indexer I have configured two inputs sources : redis and aws s3, like this :
Logs ELB API
bucket => "s3.prod.elb.logs.eu-west-1.mydomain"
prefix => "elb_api/AWSLogs/653588882345/elasticloadbalancing/"
interval => 60
region => "eu-west-1"
type => "elb_access_log"
Logs REDIS API
data_type => "list"
batch_count => 100
key => "filebeat"
host => "redis.prod.eu-west-1.mydomain.com"
In the first architechture without the redis broker, logstash was indexing my api logs from filebeat as well as elb s3 logs.
But it just stopped indexing s3 inputs without any errors. Now I just have only my api logs indexed from the redis broker.
I restarted logstash service many times without any changes, I still have only api logs in elasticsearch.
I didn't find anyway to debug s3 indexing when logstash is running as a service/daemon. I then stop the service and ran it like this :
/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/main.conf --debug
And this way, I can see logstash indexing s3 and redis inputs in real time.
When I stop the above forground command and restart the service in the background I have no more elb_logs in my elascticsearch indices.
Please can you tell me where the problem can be? Is there a way to activate s3 debuging in logstash daemon?
Thanks for help.