Is there a Beat that can read from a Kafka Topic (e.g. one that contains status from the Microservices)?

Elastic Stack Beats
1

The topology of our project is 3 Kafka nodes and 2 Cassandra nodes.
Right now I have configured the ELK Stack to output to Elastic Search.
For now we not going to use Logstash.
I have able to configure a heartbeat to ping the 3 Kafka nodes. That is not enough for us.
What Beat can I use to connect to the Kafka node directly and read a topic from it.

2

Hello, thanks for the question on Beats & Kafka. I'm not sure what beats and versions you're using, but would the Kafka module from Metricbeat meet your needs?

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kafka.html

3

For microsservices process monitoring my team had setup ELK stack with an output going directly to the Elastic Search rather than into Logstash. We are using ELK ver 7.3.2 with a Basic license and running under a Linux RedHat.

I was able to install Logstash with Kafka input and output into Elastic Search and I had also created my own topic for testing.

I have configured Logstash to see my topic and was able to successfully send a message through Kafka console command, create an index and show it in Kibana as well as my topic name in the MetricBeat.

How do I either have to make a bash/shell/config script to send a status check message to Kafka or use another way to send a status check message via Kafka and get back a status check response and use that for process monitoring.

How do I either have make a script to send a status check message to Kafka or use another way to send a status check message via Kafka and get back a status check response.

4

To read a Kafka topic directly from beats, there is a new Kafka input in Filebeat 7.4 which is expected to release early next week, which might address some of your needs. It sounds like you might be happy with Logstash now, though. I'm not sure about your last question about status checks... what kind of status check are you looking for? Is this post the kind of thing you mean? Are you looking for a way to log periodic kafka status checks to elasticsearch?

5

My team is building a platform that uses Microservices. Kafka is used to send messages for communication between MIcroservices and Cassandra.
I am not trying to figure out how to monitor network infrastructure or Kafka itself.
There are two questions I am trying to figure out a solution for.
If a Kafka's broker goes down how do I detect it and show that in Kibana?
What if a microservice goes down or is idling for a long time period?
Using ELK Stack '7.3.2'. Using 'kafka_2.12-2.1.1'

How do I visualize my JSON in Kibana showing lets say Up/Down for particular broker.

Here is my Logstash.config

# Beats -> Logstash -> Elasticsearch pipeline.
input {
  kafka {
    bootstrap_servers => "nydvalkafvm01.tradeware.com:9092"
    topics => ["boris-topic"]
    codec => json
  }
}
filter{
     json {
       source => "message"
 }
}
output {
  elasticsearch {
    action =>"index"
    hosts => ["http://nyuatalcasvm01:9200"]
    index => "boris-topic"
    workers => 1
    codec => json
    #user => "elastic"
    #password => "changeme"
  }
}
6

Yes. My team has microservices interacting with Kafka. I am able to create my own topic and send a message through Kafka Console then though Logstash send it to Elasticsearch and show in Kibana.
My question is how I can send a message to Kafka from ELK Stack.

7

You can set up beats or logstash to write to kafka using the kafka output settings: here's the kafka output for filebeat, and for logstash

8

Using ELK Stack :
How can I send a message directly to Kafka to see if a broker goes does down?
How can I send a message via Kafka to see if a process is down?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.