Thanks Jai;
SOLVED
our configurations were eerily similar, but there was one difference that got me thinking about something; I'm actually using multiple devices.
( APPLIANCE, running Filebeats ) -> (INGEST, running Kafka and Logstash) -> (ELASTICSEARCH...)
When I deployed Filebeats against my Ingest server and it's Kafka instance I had an initial error because the discovery forced Kafka to advertise it's hostname but Filebeats couldn't do a DNS lookup in my test environment. The DNS look up seemed to be due to a default setting that gets the hostname in Kafka. So I changed the server.properties file in Kafka for it to advertise on it's IP and this allowed filebeat to hook into kafka and work.
When I was looking at your logstash conf file it got me thinking that maybe discovery wasn't working because "localhost:9092" wasn't resolving kafka cleanly; I quickly changed the bootstrap server setting in my Logstash conf to point to the Ingest servers IP that kafka was set to advertise on instead of the loopback interface and it's all back up and running.
A couple of points on reflection:
- This was initially working, but stopped working on restart. I'd love to know why. Now that we know the fix perhaps someone can shed some light?
- Based on what I observed is there any logging we can get in Logstash that might make it clear it's not properly doing something - the only difference in the logs that I see now is that the file output filter indicates that it's opening a file for output...
Thanks to everyone that responded.
Greg