Kafka to Logstash pipeline not working

Elastic Stack Logstash
1

HI,

I have this pipeline:
Apache Webserver + Filebeats -> Logstash1 (no filters) -> Kafka -> Logstash2 (Filters) -> ES + Kibana

Via my Kafka consumer, I can see the logs going to the Kafka channel. And in Kibana I can see the heart beat being indexed in ES. But for some reason, there is no longer any logs going Kafka->LS->ES. I look at the Logstash2's log file and there are no errors, and the config is ok. I have removed all the filters and the ips and ports are good. I created a new channel in Kafka and temporarily had logs arriving in ES but then it stopped again (the Heartbeats continued).

Anyone have any idea what is going on? Thanks

2

In the second Logstash instance, replace the elasticsearch output with a simple stdout { codec => rubydebug } output so that you can see exactly what happens. Does it help? Increase the Logstash loglevel by starting it with --verbose or even --debug. Do you get any useful clues? Can you see if Logstash connects to Kafka? Et cetera.

3

Ok, so I moved my groks from Ls1 to Ls2, but the mistake being that I didn't factor in that it was grok'ing on a modified bunch of text. In my troubleshooting, I opened a Kafka producer console and noted that the text was ending up in ES, which alluded to an issue with the grok. Wish I figured that out sooner!

Thanks for your reply, I appreciate you trying to help me.