Help debugging ingestion flow Filebeat + Logstas

Hi everybody,

I have setup an BLEK (Beats, Logstash, ElasticSearch, Kibana) stack :stuck_out_tongue:

  • ES and Kibana are running in ES Cloud
  • a "cluster" of Logstash instances are running on a Kubernetes cluster
  • A service and Filebeat are running from an other kubernetes cluster.

The logstash has been deployed from Helm & Filebeat while using the AllInOne file provided by the "getting Started" page of ES doc.

At some point I have a hole in my log collection for one of my service:

The local Filebeat does not seems to complain. But all I get is filebeat saying it has metrics

The aggregator layer with Logstash does not say much more during the same time window:

Screenshot_20200817_171121_logs_logstash

The events after 00:00 seems related to an other issue that has nothing to do with this one (Complaining not beeing able to write in the index logs-xxxx-2020-08-15 which has been moved to other nodes by the LifeCyclePolicy whereas we are 2020-08-17 ...)

But the shipped-in collectionof GCP shows logs ....

Does some one have a hint or an idea where I can start looking at ?

About the config of Logstash:

input: beats
output: elasticsearch

By looking at Kubernetes, neither Filebeat or logstash pods have restarted for months ...

This gives me serious doubts about the reliability of the BLEK stack :frowning_face:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.