# Metribeats to kafka: no dashboard appear in kibana
**URL:** https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688
**Category:** Beats
**Tags:** metricbeat
**Created:** [July 14, 2022, 8:58pm UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688 "2022-07-14T20:58:35Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 14, 2022, 8:58pm UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/1 "2022-07-14T20:58:35Z")
Hello,
i have an issue with metricbeat. When i send data from metricbeat to kafka and kafka send to logstash and logstash then in kibana.
i can display the datas in kibana but i do not see the graphics displays in kibana , i only see the logs.
When i send the datas directly to elasticsearc i see the grapfics.
So i have some questions.
is it possible to display the graphics even if we use kafka, if so can you refer me to a tutorial or a guide so i can do it myself
Thank you
---
### Author:  [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)
#### Post date: [July 15, 2022, 1:32am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/2 "2022-07-15T01:32:08Z")
When you are sending the events from Logstash → Elasticsearch, are you writing to the same index as if you are sending from Metricbeat → Elasticsearch?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 15, 2022, 2:52am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/3 "2022-07-15T02:52:24Z")
Hi @emmanuel_stevens_LED
First What version of the Stack are you on, this is important as there some configuration differences.
What I usually suggest is a progression
**Step 1**
metricbeat -\> Elasticsearch
Works! This is good you said it already works.
Complete
**Step 2**
Next get this working...
Metricbeat -\> Logstash -\> Elasticsearch
There are several steps to get this to work, and you need a proper logstash config especially if you are using metricbeat modules. If you tell us what version you are one I can share a sample config.
Get that to work....
Only After you get Step 2 to work proceed to Step 3.
**Step 3**
Metricbeat - Kafka -\> Logstash -\> Elasticsearch
You will need to make sure beats is correctly writing to Kafka and that Logstash is pulling the right messages from the topc
I see people spend a lot of time try to set it all up at once and spend a lot of frustrating time... this is just my experience and advice.
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 15, 2022, 4:37am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/4 "2022-07-15T04:37:52Z")
Hello @stephenb ,
i am using version 8.3.2 for metricbeat.
Metricbeat is correctly writing to kafka, i can see the logs coming in the console as a consumer. Logstash send everything to kibana, i can see these logs in **Observability --logs--stream**
I want to display proper graphic and maybe put some filter to display the logs in a better way
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 15, 2022, 4:51am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/5 "2022-07-15T04:51:06Z")
I do not send the logs directly from metric beats to elasticsearch. I send them to kafka and aftre logstash take it from kafka and send the log to elastiocsearch. My configurations are base on what is on the web site, nothing more.
---
### Author:  [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)
#### Post date: [July 15, 2022, 5:03am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/6 "2022-07-15T05:03:03Z")
Did you index your Metricbeat events to `logs` index/data stream? Because the log stream UI defaults to either `filebeat-*` or `logs-*`, so either you changed this setting or you are indexing into the wrong index.
And based on the same screenshot, it seems that the JSON are not parsed correctly. Did you use the JSON codec in Logstash when you consume the events from Kafka?
May be worthwhile to share your Logstash pipeline configurations.
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 15, 2022, 5:14am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/7 "2022-07-15T05:14:20Z")
Here is my logstash config
```auto
input { kafka {
bootstrap_servers => "192.168.208.160:9092"
topics => ["test", "metricbeat"]
}
}
#output {
# stdout {}
# }
output {
elasticsearch {
hosts => ["https://192.168.208.135:9200","https://192.168.208.136:9200"]
cacert => '/etc/logstash/elasticsearch-ca.pem'
user => 'elastic'
password => '+5lwdwgSNuMF_8aJKQlD'
}
}
```
---
### Author:  [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)
#### Post date: [July 15, 2022, 5:31am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/8 "2022-07-15T05:31:32Z")
You may want to remove your `elastic` user password, and you shouldn't be using `elastic` user for ingestion.
Having said that, the `index` is missing from the `elasticsearch` output, so most likely Logstash is indexing into `logstash` index by default.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 15, 2022, 5:32am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/9 "2022-07-15T05:32:31Z")
Hi @emmanuel_stevens_LED
Seeing as you're going to ignore my advice 🙂 which is okay but I'll only be able to help a bit then.
Seeing as you are skipping step 2 above you can try this..
Your logstash configuration is not correct for the output ... This assumes your ran
`metricbeat setup -e`
when you first set up
Also this will not work if you setup modules... And need pipelines ... But for base metricbeat it will probably work
```auto
elasticsearch {
hosts => "http://localhost:9200"
user => "elastic"
password => "password"
manage_template => false
index => "metricbeat-8.2.3"
action => "create"
}
```
You might also want to look learn [here](https://discuss.elastic.co/t/logstash-not-indexing-date-to-the-filebeat-write-index-for-data-stream/306360/2)
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 15, 2022, 5:47am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/10 "2022-07-15T05:47:14Z")
@stephenb
I already had everything setup on my vmware. I already had a Kafka cluster running.
I am going to test what you said.
One more question, do i have to setup a conf file for each topic. I send metricbeat data on the metricbeat topic and send filebeat data on test topic. Do you have a suggestion or a link to share ?
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 15, 2022, 5:51am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/11 "2022-07-15T05:51:06Z")
thank you for your answers @hendry.lim , it is really helpful
For the user, when I use the logstash\_system user to ingest data, it gave me an error not authorized. Since I am testing, so I use the elastic user.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 15, 2022, 6:15am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/12 "2022-07-15T06:15:01Z")
My point is not comment on your final architecture, which is fine.. It is how to build it up and understand each component.
I'm not the Kafka expert perhaps someone else is..
I would definitely review the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html)
> [@emmanuel\_stevens\_LED](#):
>
> One more question, do i have to setup a conf file for each topic. I send metricbeat data on the metricbeat topic and send filebeat data on test topic.
No I think you can do one with 2 inputs and with conditional output based on tags...
Or
You can create 2 separate as well and name them in the pipelines.yml file
Your choice
---
### Author:  [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)
#### Post date: [July 15, 2022, 6:36am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/13 "2022-07-15T06:36:19Z")
You can consume from multiple topics with one `kafka` input and add conditional output based on the topic/beat name, for example.
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 21, 2022, 5:06am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/14 "2022-07-21T05:06:47Z")
Hello, I confugured the logstash to send the logs to my elk cluster, in that case i remove the kafka cluster so metricbeat communicate directly to logstash .
I putted the index in the logstash conf (output plugin ) as shown in your answer.
The dashboard still not receive the datas. Maybe i supected that kibana is pointing to anothe index.
Do you have a documents that can explain clearly what to do , or maybe a tutorial. I really need to use those dashboards
Thx
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 21, 2022, 5:41am UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/15 "2022-07-21T05:41:15Z")
Read this... And follow the steps
Where it's say to clean up that means completely remove any / All metricbeat on indices and or data streams
> [@Metricbeat to Logstash to ElasticSearch - Cannot See Any Hosts Defined, But Data Is Definitely Coming In](https://discuss.elastic.co/t/metricbeat-to-logstash-to-elasticsearch-cannot-see-any-hosts-defined-but-data-is-definitely-coming-in/275715/2):
>
> Hi @jthart Welcome to the community apologies that you're having some struggles getting this set up Perhaps we can help. Assuming you want to run an architecture like this Metricbeat (1 to Many) -\> Logstash -\> Elasticsearch Basically using Logstash as a collect and pass through Here is my recommendation try to resist the urge to make this more complex. Do not try to manually load index templates dashboards anything else follow the quick start / basic setup. Clean everything up we're star…
Gives you step by step...
And with the new datastreams logstash should look like this.
```auto
input {
beats {
port => 5044
}
}
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => "http://localhost:9200"
pipeline => "%{[@metadata][pipeline]}"
user => "elastic"
password => "password"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
action => "create"
}
} else {
elasticsearch {
hosts => "http://localhost:9200"
user => "elastic"
password => "password"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
action => "create"
}
}
}
```
These are the steps / configuration that work / work for others.
Also, you're logstash config is not correct. You're not looking close at the elasticsearch output. You need to write to the data stream so it uses to correct mapping and then rolls over with the island. When you added the date to it, it wasn't doing that correctly.
Get that working then you can put Kafka back in.
---
### Author:  [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)
#### Post date: [July 24, 2022, 9:51pm UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/16 "2022-07-24T21:51:15Z")
hello,
I hadone what you said.
When i send the data corectly from metricbeat to elasticsearch, i have all the grafic. when i include logstash I change the logstash output to what you said and i send the data. i have some statistics but not all.
I also have an error message.
Here is a jason
```auto
{
"took": 16,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 1,
"skipped": 1,
"failed": 1,
"failures": [
{
"shard": 0,
"index": "metricbeat-1",
"node": "xD2iV5ovToG9T0iiO4pIqQ",
"reason": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default.
Please use a keyword field instead. Alternatively, set fielddata=true on [system.network.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
]
},
"hits": {
"total": 0,
"max_score": 0,
"hits": []
}
}
```
My other question. do we have to go through that process for each beat ?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 24, 2022, 9:56pm UTC](https://discuss.elastic.co/t/metribeats-to-kafka-no-dashboard-appear-in-kibana/309688/17 "2022-07-24T21:56:30Z")
> [@emmanuel\_stevens\_LED](#):
>
> ` "index": "metricbeat-1",`. \