# Data stream logs not showing in discover
**URL:** https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662
**Category:** Kibana
**Tags:** datastreams
**Created:** [July 26, 2022, 3:07pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662 "2022-07-26T15:07:18Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@metalaarif](https://discuss.elastic.co/u/metalaarif)
#### Post date: [July 26, 2022, 3:07pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/1 "2022-07-26T15:07:18Z")
So I have a setup where my rabbitmq send logs to logstash and then to elasticsearch and kibana.
This is what I have setup in my /etc/logstash/conf.d/rabbitmq.conf
```auto
input {
rabbitmq {
host => "0.0.0.0"
port => 5672
durable => true
exchange => "logs"
exchange_type => "fanout"
user => "rabbit"
password => "password"
queue => "application"
tags => ["rabbitmq"]
}
}
output {
if "rabbitmq" in [tags] {
elasticsearch {
hosts => "https://localhost:9200"
user => "elastic"
password => "password"
ssl_certificate_verification => false
index => "applog-%{[extra][tags][client_name]}-%{channel}-%{+yyyy.MM}"
}
}
}
```
With this setup everything is fine, I can see my logs in discover and everything is as it should be. However, when I start working on my ILM. I create my policy and my index template - As soon as I enable data stream I don't see logs in discover and the datastream filesize are 225 bytes. I have no idea why it does that. If I remove datastream then I can see logs but policy doesn't work.
Hope someone can help me.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 26, 2022, 4:39pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/2 "2022-07-26T16:39:10Z")
Hi @metalaarif Welcome to the community
What is the name of your data stream?
your `index` in filebeat needs to point to the data stream write alias... so the data is writing to the stream not a concrete index.
`index => "my-data-stream-name"`
See example [here](https://www.elastic.co/guide/en/elasticsearch/reference/8.3/set-up-a-data-stream.html#create-data-stream)
then ILM will work etc... you can even test rollover with
`POST my-data-stream/_rollover`
It looks like you may be trying to create a data stream for each client... if so ... you will need to create a datastream for each client **before** you start indexing... the data... not sure if that is what you are trying to do....
---
### Author:  [@metalaarif](https://discuss.elastic.co/u/metalaarif)
#### Post date: [July 26, 2022, 4:58pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/3 "2022-07-26T16:58:35Z")
Hi Stephenb,
Thanks for replying and glad to be in this community, I am not using filebeat and my rabbitmq generates all the logs in this **applog-%{[extra][tags][client\_name]}-%{channel}-%{+yyyy.MM}** format and pushes it to logstash and I can clearly see the logs.
I am not sure what you mean by data stream name but I create it when I was creating index template. Assuming index template name is data stream which I don't think it is, then the name should it be like this
```auto
index => applog-template%{[extra][tags][client_name]}-%{channel}-%{+yyyy.MM}"
```
or are you saying that this is how I need to create a data stream and ignore clicking the button in index template
```auto
PUT applog-data-stream/_bulk
```
Our logs generate in applog-\* format so in this scenario do I still need to create for each client?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 26, 2022, 5:24pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/4 "2022-07-26T17:24:20Z")
Apologies yes logstash... not filebeat.
> [@metalaarif](#):
>
> ignore clicking the button in index template
Note sure what that means...
Please share your complete index template and ILM policy
```auto
GET _index_template/my-index-template
GET _ilm/policy/my-ilm-policy
```
> [@metalaarif](#):
>
> Our logs generate in applog-\* format so in this scenario do I still need to create for each client
I think you need to look at all the data stream setting in the logstash elasticsearch output... see [here](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream)
The index needs to be the data stream name.
`"index => my-data-stream-name"`
If you put the client name in the index name you are going to add a lot of complexity...
Perhaps you want to use namespace in the data stream... I think you are mixing concepts a bit, would need to think about that..
Perhaps you could use the `data_stream_namespace`... [here](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream_namespace)
I would probably start without the client name... get it working... (as long as you have a client field you will always be able to filter)
THEN I would probably work towards using the name space..
A data stream for each client will probably be difficult unless you are saying you have a very small number of clients... you would need to automate alot I suspect.
Our you can just go back to your daily indices ... that you have working but that may create many small indices ...
---
### Author:  [@metalaarif](https://discuss.elastic.co/u/metalaarif)
#### Post date: [July 26, 2022, 7:51pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/5 "2022-07-26T19:51:36Z")
This is my index\_template and I have removed data stream from index\_template temporarily because the policy was working but I couldn't see the logs in discover.
```auto
{
"index_templates": [
{
"name": "applog-temp",
"index_template": {
"index_patterns": [
"applog-*"
],
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "applog-policy",
"rollover_alias": "applog"
},
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"refresh_interval": "5s",
"number_of_shards": "1",
"number_of_replicas": "0"
}
}
},
"composed_of": []
}
}
]
}
```
and my \_ilm/policy and please ignore the min\_age that's just for testing
```auto
{
"applog-policy": {
"version": 11,
"modified_date": "2022-07-26T17:06:50.644Z",
"policy": {
"phases": {
"warm": {
"min_age": "10h",
"actions": {
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "15h",
"actions": {
"set_priority": {
"priority": 0
}
}
},
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "1d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
},
"data_streams": [],
"composable_templates": [
"applog-temp"
]
}
}
}
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 26, 2022, 8:05pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/6 "2022-07-26T20:05:54Z")
> [@metalaarif](#):
>
> but I couldn't see the logs in discover.
Did you create a Data View.. .you need that to see logs in Discover.. you may have already had it working
Kibana -\> Stack Management -\> Data View -\> Create Data View
if you did
`GET _cat/indices`
did you see the data stream indices.. .they start with `.ds-...`
---
### Author:  [@metalaarif](https://discuss.elastic.co/u/metalaarif)
#### Post date: [July 26, 2022, 8:47pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/7 "2022-07-26T20:47:31Z")
Yes, like you mentioned I whenever I hit that command in Dev Tools
```auto
GET cat/indices
```
I can see all of them listed. The policy seems like working fine as I can see it go from Hot to Warm to Delete. The rollover worked fine too and I could clearly see **.ds-xxxxxx-000001, -000002, -00003** etc. being generated however, I could never see any logs in Kibana ==\> discover.
I don't know why that was happening but as soon as I enable data stream in index\_template the total size of indices are just 225 byte and 0 docs.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 26, 2022, 9:04pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/8 "2022-07-26T21:04:07Z")
> [@metalaarif](#):
>
> **.ds-xxxxxx-000001, -000002, -00003** etc. being generated however,
By Defintion those are backing indices for a data stream
> [@metalaarif](#):
>
> but as soon as I enable data stream in index\_template
You mean this line
```auto
"data_stream": { },
```
> [@metalaarif](#):
>
> I don't know why that was happening but as soon as I enable data stream in index\_template the total size of indices are just 225 byte and 0 docs.
Sorry I have lost track ... to many moving parts...
if you just want to try indices (not data stream)
If you take what you have above without the data stream
Cleanup up
Then bootstrap the write alias index
```auto
PUT
{
"aliases": {
"applog": {
"is_write_index": true
}
}
}
```
then set in logstash output
`index => "applog"`
It should all work
You will still need to create a Data View in Kibana to see that data
---
### Author:  [@metalaarif](https://discuss.elastic.co/u/metalaarif)
#### Post date: [July 26, 2022, 9:20pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/9 "2022-07-26T21:20:55Z")
Yes, I did create data view and I couldn't see them at all.
But I will try what you've mentioned and I will change my `index => applog`
> Then bootstrap the write alias index
```auto
PUT
{
"aliases": {
"applog": {
"is_write_index": true
}
}
}
```
This is something I have not tried. I'll try this and get back to you.
Thank you so much for swift reply.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 26, 2022, 9:23pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/10 "2022-07-26T21:23:55Z")
That ^^^^ is straight up Elastic Magic 🙂
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 26, 2022, 10:01pm UTC](https://discuss.elastic.co/t/data-stream-logs-not-showing-in-discover/310662/11 "2022-07-26T22:01:52Z")
@metalaarif
**I found the problem with the logstash and data streams DARN .**.. I should have recognized it... as I have written on it before... DOH!!! Apologies... and this is a bit of a bug right now..
Everything you did in the beginning was right (except the client id) but you need to add when using the `index` directive...
` action => "create"`
in the elasticsearch output section. Basically logstash was failing to write the data ... because Data Stream _only_ allow create and you have to set it manually.
```auto
output {
if "rabbitmq" in [tags] {
elasticsearch {
hosts => "https://localhost:9200"
user => "elastic"
password => "password"
ssl_certificate_verification => false
index => "applog"
action => "create"