I need someone to please verify my simple config because I'm not seeing the logs in Kibana like I thought. It's very simple; here's what I have. I would think the container I run on the daemon; whatever it logs would end up in Kibana.
Server1 - docker daemon configured with log driver to udp gelf messages to Server2
Server2 - logstash, elasticsearch, kibana all running and communicating with each other.
I'm running RHEL 7 on both servers(VM's).
GELF+logstash ->I verified all the communication SEEMS to be correct. I can manually send a UDP message from server1 and have the logs in logstash on server2 complain it can't parse the UDP message (because I don't send the manual UDP message in GELF format). But that at least tells me server1 and 2 can interact via UDP on the port I specified.
elasticsearch -> I can hit that and get back that generic JSON response in the brower
kibana -> I have the index to look for logstash messages and at one point I saw some logs in kibana. I haven't changed anything but I don't see any logs.
However I don't see any plain docker logs in kibana (whatever those would be; I've never seen any, not even daemon start-up messages) and when I run a container on server1 whatever it logs never ends up in server2's kibana.
Can someone at least read this and verify I have the basic setup of docker and gelf etc correct if all my ports and routes are working?
docker config =
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --log-driver gelf --log-opt gelf-address=udp://server2:8202
logstash config (on server2):
input {
gelf { port => 8202 }
}
output {
elasticsearch { hosts => ["localhost:8200"]}
}