Hi all,
I've put "Kibana" as the category for the post, but realistically it could be any part of the Elastic stack that I've misconfigured.
Elastic stack structure
Azure-hosted VMs running the following; part of the same IP range as the DCOS platform:
- Elastic-master
- Elastic-data-1
- Elastic-data-2
- Logstash
- Kibana
Application infrastructure
- Master Nodes (Azure-hosted Linux VMs) running DCOS container orchestration platform
- Agent Nodes (Azure-hosted Linux VM Scale Set) running containers, orchestrated by DCOS
- Application comprised of NodeJS microservices each running as a container
- Microservices log to container's STDOUT
- Containers' STDOUT entries are stored in log files on whichever Agent Node they're running on
Log-scraping
- Filebeats container running on each Agent Node
- Scrapes logs from Agent Node
- Passes to Logstash
- Logstash pipeline filters/categorises log entries and passes to Elasticsearch
This setup works as intended in our Development environment. In Kibana, I can see all of our containers' logs/log entries and they're parsed correctly.
I've just recreated the ELK cluster in our Pre-Production environment and encountered the following problems in Kibana:
- Not all log entries appear to be coming through; I can only see 5 of our ~20 microservices
- If I click on "Logs", I see the error "Looks like you don't have any logging indices." yet if I go to "Management" then "Elasticsearch Index Management", I can see my indices with green health status
In an attempt to diagnose these problems, I would like to know:
- How can I be certain that all of the Elastic nodes are working together and processing data?
- In my Logstash pipeline, should I be specifying the IP addresses of all of my Elastic nodes (Master and 2x Data) or just the Master? Currently I'm just specifying
hosts: [10.0.0.5]which is the IP of the Master - Is there a way that I can see the log entries that Logstash is sending to Elasticsearch, and a way that I can see the output that Kibana is accessing from Elasticsearch?
Thanks in advance