# Using Elastic stack with Kafka for reading system metrics

**URL:** <https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032>\
**Category:** Logstash\
**Created:** [February 19, 2019, 1:55pm UTC](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032 "2019-02-19T13:55:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![astrovj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/astrovj/32/81914_2.png) [@astrovj](https://discuss.elastic.co/u/astrovj)\
**Post date:** [February 19, 2019, 1:55pm UTC](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032/1 "2019-02-19T13:55:22Z")

</div>

Hello,  
Below is the flow of my Elastic stack (setup1)  
metricbeat (system module) -\> kafka cluster -\> logstash -\> elasticsearch -\> kibana

In kibana I get the metrics in form of a single message like below

**message** :{"@timestamp":"2019-02-19T11:32:07.531Z","@metadata":{"beat":"metricbeat","type":"doc","version":"6.6.0","topic":"Nutanix2"},"event":{"dataset":"system.process","duration":21110912},"system":{"process":{"pid":301,"username":"root","cpu":{"total":{"value":2.31066e+06,"pct":0.001,"norm":{"pct":0.0003}},"start\_time":"2018-11-12T06:04:08.000Z"},"memory":{"size":0,"rss":{"pct":0,"bytes":0} }

If I send the metrics from metricbeat (system module) -\>elastic search -\>kibana , the metrics come in form of (field:value) rather than as a single message and I can also use dasboards for visualization. For eg.

**@timestamp** :February 19th 2019, 13:44:36.895  
**beat.name** :ecs-e976.novalocal  
**beat.hostname** :ecs-e976.novalocal  
**beat.version** :6.2.3  
**metricset.module** :system **metricset.rtt** :73 **metricset.name** :network **system.network.name** :eth0 system.network.in.bytes:2.188GB **system.network.in.packets** :19,695,275 **system.network.in.errors** :0 **system.network.in.dropped** :0 **system.network.out.dropped** :0

**My question is can I do something in my setup1 which will allow kibana to get the metrics in form of field:value. (other than writing grok filter).**

**My metricbeat output settings:**  
output.kafka:

# initial brokers for reading cluster metadata

hosts: ["192.168.1.63:9092", "192.168.1.61:9092", "192.168.20.2:9092"]

# message topic selection + partitioning

topic: 'Nutanix2'  
partition.round\_robin:  
reachable\_only: false

required\_acks: 1  
compression: gzip  
max\_message\_bytes: 1000000

**My Logstash conf file**  
input {  
kafka {  
id =\> "my\_id"  
bootstrap\_servers =\> "192.168.1.63:9092, 192.168.1.61:9092, 192.168.20.2:9092"  
topics =\> ["Nutanix2"]  
}  
}

output {  
elasticsearch {  
hosts =\> ["192.168.1.231:9200"]  
index =\> "Nutanix2-%{+YYYY.MM.dd}"  
}

```
file {
		path => /tmp/lokaf.out
		codec => line { format => "custom format: %{message}"}

```

}  
}

Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 19, 2019, 2:04pm UTC](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032/2 "2019-02-19T14:04:38Z")

</div>

Either add 'codec =\> json' to the kafka input, or add

```
 filter { json { source => "message" } }
```

---

<div class="post-metadata">

**Author:** ![astrovj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/astrovj/32/81914_2.png) [@astrovj](https://discuss.elastic.co/u/astrovj)\
**Post date:** [February 20, 2019, 8:43am UTC](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032/3 "2019-02-20T08:43:19Z")

</div>

Thanks a ton. Appreciate it. 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 20, 2019, 8:43am UTC](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032/4 "2019-03-20T08:43:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.