# How to use Metricbeat with Kafka **URL:** \ **Category:** Beats\ **Created:** [July 26, 2017, 7:36am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585 "2017-07-26T07:36:28Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [July 26, 2017, 7:36am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/1 "2017-07-26T07:36:28Z")
Hello All, i am using Filebeat version 5.5 to send logs to kafka topic. But I am not able to send the TOP command logs. Is there any way to send TOP command logs to Kafka? Thank You ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [July 26, 2017, 9:23am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/2 "2017-07-26T09:23:02Z")
Not sure what you mean by the TOP command logs. Are you interested in CPU and memory metrics? Best would be to just use Metricbeat in that case. ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [July 26, 2017, 9:24am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/3 "2017-07-26T09:24:52Z")
Yes, exactly I want to store CPU ,Memory and Disk Utilization logs to Kafka. ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [July 26, 2017, 9:37am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/4 "2017-07-26T09:37:21Z")
I'd recommend using Metricbeat with the Kafka output. ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [July 26, 2017, 9:43am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/5 "2017-07-26T09:43:45Z")
Thanks for quick reply. I want to collect data of system resource utilization, such as which process consumes CPU, memory. And also how much total cpu, memory & disk is utilize and how much is free. Can you please help in this @tudor ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [July 26, 2017, 11:43am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/6 "2017-07-26T11:43:08Z")
> [@rahul01](#): > > I want to collect data of system resource utilization, such as which process consumes CPU, memory. > And also how much total cpu, memory & disk is utilize and how much is free. This is what [Metricbeat's](https://www.elastic.co/products/beats/metricbeat) system [module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-system.html) does. Give it a try. ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [July 26, 2017, 2:15pm UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/7 "2017-07-26T14:15:04Z")
Thanks for reply. I gave a try and got output as needed, but I didn't got output for disk space usage like how much disk is used and how much it is free. It gave me output for disk I/O only. ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [July 27, 2017, 8:58am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/8 "2017-07-27T08:58:32Z")
The [fileseystem](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-filesystem.html) metricset should include that information. Look for the `system.filesystem.available`, for example. ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [July 28, 2017, 9:55am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/9 "2017-07-28T09:55:45Z")
Thanks @tudor , My issue is solved 🙂 ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [July 31, 2017, 7:16am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/10 "2017-07-31T07:16:53Z")
All values are shown in bytes, So Is there any way to get value in MB or GB? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [August 1, 2017, 1:13am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/11 "2017-08-01T01:13:58Z")
If you are using Kibana and you loaded the provided index patterns using the `import_dashboards` tool then the fields containing bytes will be rendered using KB/MB/GB/TB depending on the value. Kibana uses [http://numeraljs.com/](http://numeraljs.com/). [https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-sample-dashboards.html](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-sample-dashboards.html) ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [August 2, 2017, 6:11am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/12 "2017-08-02T06:11:58Z")
Thanks @andrewkroh I got it. I set 3 modules (load, memory & fsstat) and got output for all 3 modules in 3 sets. So my sir said that output is too long and told me to find that how to minimize output. [ { "@timestamp": "2017-07-31T10:22:30.825Z", "beat": { "hostname": "kafkatest", "name": "kafkatest", "version": "5.5.1" }, "metricset": { "module": "system", "name": "memory", "rtt": 699 }, "system": { "memory": { "actual": { "free": 1677877248, "used": { "bytes": 230150144, "pct": 0.1206 } }, "free": 1040068608, "swap": { "free": 1073737728, "total": 1073737728, "used": { "bytes": 0, "pct": 0 } }, "total": 1908027392, "used": { "bytes": 867958784, "pct": 0.4549 } } }, "type": "metricsets" }, { "@timestamp": "2017-07-31T10:22:30.825Z", "beat": { "hostname": "kafkatest", "name": "kafkatest", "version": "5.5.1" }, "metricset": { "module": "system", "name": "fsstat", "rtt": 936 }, "system": { "fsstat": { "count": 30, "total\_files": 2640058, "total\_size": { "free": 21922209792, "total": 24112676864, "used": 2190467072 } } }, "type": "metricsets" }, { "@timestamp": "2017-07-31T10:22:30.826Z", "beat": { "hostname": "kafkatest", "name": "kafkatest", "version": "5.5.1" }, "metricset": { "module": "system", "name": "load", "rtt": 812 }, "system": { "load": { "1": 0, "5": 0.01, "15": 0.05, "norm": { "1": 0, "5": 0.0006, "15": 0.0031 } } }, "type": "metricsets" } ] Can this output be shorten? ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [August 2, 2017, 8:56am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/13 "2017-08-02T08:56:22Z")
You can use the `drop_fields` processor to remove unwanted fields. See the [docs](https://www.elastic.co/guide/en/beats/metricbeat/5.5/drop-fields.html) here. ---
**Author:** ![rahul01](https://avatars.discourse-cdn.com/v4/letter/r/b5e925/32.png) [@rahul01](https://discuss.elastic.co/u/rahul01)\ **Post date:** [August 9, 2017, 10:24am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/14 "2017-08-09T10:24:57Z")
drop\_fields processor will only exclude module data. Here is kind of example for output I want, [ { "@timestamp": “2017-07-31T10:22:30.825Z”, “beat”: { “hostname”: “kafkatest”, “name”: “kafkatest”, “version”: “5.5.1” }, “metricset”: { “module”: “system”, “name”: “memory”, “rtt”: 699 }, . . . “metricset”: { “module”: “system”, “name”: “fsstat”, “rtt”: 936 }, . . . “metricset”: { “module”: “system”, “name”: “load”, “rtt”: 812 }, . . . ] Just want to exclude this lines which are coming for each metricset module { "@timestamp": “2017-07-31T10:22:30.826Z”, “beat”: { “hostname”: “kafkatest”, “name”: “kafkatest”, “version”: “5.5.1” }, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2017, 7:36am UTC](https://discuss.elastic.co/t/how-to-use-metricbeat-with-kafka/94585/15 "2017-08-16T07:36:31Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.