Investigating why all logs aren't visible in Kibana

Elastic Stack Logstash
1

Hi all,

I've put "Kibana" as the category for the post, but realistically it could be any part of the Elastic stack that I've misconfigured.

Elastic stack structure
Azure-hosted VMs running the following; part of the same IP range as the DCOS platform:

  • Elastic-master
  • Elastic-data-1
  • Elastic-data-2
  • Logstash
  • Kibana

Application infrastructure

  • Master Nodes (Azure-hosted Linux VMs) running DCOS container orchestration platform
  • Agent Nodes (Azure-hosted Linux VM Scale Set) running containers, orchestrated by DCOS
  • Application comprised of NodeJS microservices each running as a container
  • Microservices log to container's STDOUT
  • Containers' STDOUT entries are stored in log files on whichever Agent Node they're running on

Log-scraping

  • Filebeats container running on each Agent Node
  • Scrapes logs from Agent Node
  • Passes to Logstash
  • Logstash pipeline filters/categorises log entries and passes to Elasticsearch

This setup works as intended in our Development environment. In Kibana, I can see all of our containers' logs/log entries and they're parsed correctly.

I've just recreated the ELK cluster in our Pre-Production environment and encountered the following problems in Kibana:

  • Not all log entries appear to be coming through; I can only see 5 of our ~20 microservices
  • If I click on "Logs", I see the error "Looks like you don't have any logging indices." yet if I go to "Management" then "Elasticsearch Index Management", I can see my indices with green health status

In an attempt to diagnose these problems, I would like to know:

  • How can I be certain that all of the Elastic nodes are working together and processing data?
  • In my Logstash pipeline, should I be specifying the IP addresses of all of my Elastic nodes (Master and 2x Data) or just the Master? Currently I'm just specifying hosts: [10.0.0.5] which is the IP of the Master
  • Is there a way that I can see the log entries that Logstash is sending to Elasticsearch, and a way that I can see the output that Kibana is accessing from Elasticsearch?

Thanks in advance

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.