# Data not getting index they are getting \_ignored when using logstash
**URL:** https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058
**Category:** Elastic Search
**Created:** [June 10, 2025, 2:19pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058 "2025-06-10T14:19:12Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@any\_name](https://discuss.elastic.co/u/any_name)
#### Post date: [June 10, 2025, 2:19pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/1 "2025-06-10T14:19:12Z")
ELK version 8.17.4
Initially, data sent directly to Elasticsearch was visible in Kibana Discover. However, after disabling direct Elasticsearch output and routing the data through Logstash, the data is no longer appearing in Discover.
pipeline
```auto
input {
beats {
port => 5044
}
}
output {
if [agent][type] == "metricbeat" {
elasticsearch {
hosts => [""]
data_stream => true
data_stream_type => "metrics" # Use "logs" for log data
data_stream_dataset => "metricbeat"
data_stream_namespace => "default"
user => ""
password => ""
ssl => true
ssl_certificate_verification => true
cacert => "/etc/logstash/certs/http_ca.crt"
}
}
}
```
[@everyone](https://discuss.elastic.co/groups/everyone)
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 10, 2025, 2:42pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/2 "2025-06-10T14:42:05Z")
> [@any\_name](#):
>
> However, after disabling direct Elasticsearch output and routing the data through Logstash, the data is no longer appearing in Discover.
Are you using the same data stream name?
This is an mapping error, it means that you are receiving documents where the value of the `system.uptime` field is conflicting with the mapping defined in the template.
---
### Author:  [@any\_name](https://discuss.elastic.co/u/any_name)
#### Post date: [June 13, 2025, 8:02am UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/3 "2025-06-13T08:02:39Z")
hi @leandrojmp
I'm using the same name for the data stream. pls refer the below image for same.
I checked the data again after a few days — the `system.uptime` metrics are getting indexed now. However, the `system.filesystem` data is currently being ignored.
Previously, the `system.filesystem` data was being indexed properly, and I had even created a dashboard based on it. But now, it's no longer working.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 13, 2025, 12:34pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/4 "2025-06-13T12:34:53Z")
> [@any\_name](#):
>
> I'm using the same name for the data stream. pls refer the below image for same.
From what you shared you are not using the same datastream, in your logstash output you have it configured to write into a datastream named `metrics-metricbeat-default` and this naming pattern is not used by any tool from elastic.
When you use metricbeat to send data directly to Elasticsearch it writes into a datastream named `metricbeat-version`, which is what you had before as `metricbeat-8.17.4`, when you use elastic agent it would write into a datastream per metric type, like `metrics-system.load-default`.
You need to change the elasticsearch output in your Logstash configuration to use this:
```auto
elasticsearch {
hosts => [""]
data_stream => true
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
action => "create"
user => ""
password => ""
ssl => true
ssl_certificate_verification => true
cacert => "/etc/logstash/certs/http_ca.crt"
}
```
This is mentioned here in the [documentation](https://www.elastic.co/docs/reference/beats/filebeat/logstash-output#_accessing_metadata_fields).
---
### Author:  [@any\_name](https://discuss.elastic.co/u/any_name)
#### Post date: [June 30, 2025, 5:25am UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/5 "2025-06-30T05:25:02Z")
Hi @leandrojmp
i have tried the step you mention but faced with below error.
```auto
[2025-06-16T13:55:33,583][ERROR][logstash.outputs.elasticsearch] Invalid data stream configuration, the following parameters are not supported: {"index"=>"%{[@metadata][beat]}-%{[@metadata][version]}"}
[2025-06-16T13:55:33,584][ERROR][logstash.javapipeline] Pipeline error {:pipeline_id=>"main", :exception=>#, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.12-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:76:in `check_enabled_data_stream_config!'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.12-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:59:in `check_data_stream_config!'",
```
I removed `"data_stream=true"`, and it worked — indices are now being created successfully. Metricbeat is using the default data stream (e.g., `metricbeat-8.17.4`), which is good. However, there's still an issue: instead of creating indices for the current date, it's creating them with the previous day's date.
eg.
creating
> .ds-metricbeat-8.17.4-2025.06.29-000023
expected
> .ds-metricbeat-8.17.4-2025.06.30-000023
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 30, 2025, 11:37am UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/6 "2025-06-30T11:37:00Z")
> [@any\_name](#):
>
> However, there's still an issue: instead of creating indices for the current date, it's creating them with the previous day's date.
In which timezone are you? Logstash times is always in UTC and cannot be changed.
---
### Author:  [@any\_name](https://discuss.elastic.co/u/any_name)
#### Post date: [June 30, 2025, 12:12pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/7 "2025-06-30T12:12:05Z")
India Standard Time (GMT+5:30)
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [June 30, 2025, 1:35pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/8 "2025-06-30T13:35:12Z")
> [@any\_name](#):
>
> India Standard Time (GMT+5:30)
So, this is expected, you are ahead of UTC, all dates in Logstash and Elasticsearch are in UTC.
This cannot be changed.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 30, 2025, 1:56pm UTC](https://discuss.elastic.co/t/data-not-getting-index-they-are-getting-ignored-when-using-logstash/379058/9 "2025-06-30T13:56:32Z")
Hi @any_name
As @leandrojmp indicated the backing indices names are generated via UTC time.
Curious, What actual problem are you trying to solve?
You should not be trying to create any "logic" based on the data stream backing indices names? If it is just "new" to you ... this is the way it works 🙂