Can filebeat monitoring send data to Kafka?

Elastic Stack Beats
1

As per the monitoring options for Filebeat given here, the monitoring stats can be send to Elasticsearch.
But I would like the data to be sent to Kafka. Is this possible?.

2

I also tried to mimic Elasticsearch using Logstash in the filebeat configuration as below:

monitoring:
enabled: true
elasticsearch:
hosts: ["http://localhost:5050"]

My logstash configuration for this was below:

input {
http {
host => "0.0.0.0"
port => "5050"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "filebeat_monitoring_logs-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

But this also didnt work.
Any idea of how to collect monitoring data from filebeat to Kafka/Logstash?

3

Yes take a look here

FIXED : I put the wrong link first time...

4

@stephenb
I think what you have given is the normal data forwarding configuration to Kafka.
What I am actually looking for is the monitoring stats data which is sent by the filebeat, that is to be forwarded to Kafka/Logstash. You can see the details here

This data was what I was not able to send to Kafka/Logstash from Filebeat.

5

@arunpmohan
Ahh I see apologies... hmm... yes I don't see and obvious way... at the moment I will ask a question...

Perhaps you could try this method
https://www.elastic.co/guide/en/beats/filebeat/current/monitoring-metricbeat-collection.html

and ship the metricbeat output to Kafka ...

6

@stephenb
Hmm.. I have 2 questions

  1. we are using version 6.6. This is available only from 7.3 it seems. So will it work?
  2. Also, it is tagged "xpack", so is it an xpack only feature?
7

@stephenb
Can you please get back on the previous questions?.

8

So i do not expect it will work in 6.6 as that is explicitly 7.3 and beyond per the docs

"In 7.3 and later, you can use Metricbeat to collect data about Filebeat and ship it to the monitoring cluster..."

As for checking around, no one I spoke to had tried that yet, and I do not have a proper test harness to try that so you would need to try it on your own.

Per the "XPack" that means that it is not part of the OSS distribution in this case it is part of the "Basic / Default" distribution which is free to use.

You can see what is part of OSS, Basic, Gold, Platinum here

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.