# Logstash won't send data to ES after re-create ES Cluster(Kafka Input)
**URL:** https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230
**Category:** Logstash
**Created:** [December 1, 2022, 9:47am UTC](https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230 "2022-12-01T09:47:28Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@da-head0](https://discuss.elastic.co/u/da-head0)
#### Post date: [December 1, 2022, 9:47am UTC](https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230/1 "2022-12-01T09:47:29Z")
Hello,
ELK version : 8.4.2
Environment: Logstash in EKS, ECK Elasticsearch
Hello, I used to send data to Elasticsearch with these config.
```auto
input {
kafka {
bootstrap_servers => "xx:9094, xx2:9094"
topics_pattern => ".*"
consumer_threads => 4
codec => "json"
decorate_events => "basic"
group_id => "kafka"
client_id => "logstash"
security_protocol => "SSL"
ssl_truststore_location => "/logstash/xx"
enable_auto_commit => "true"
partition_assignment_strategy => "round_robin"
auto_commit_interval_ms => 1000
}
output {
if [kafka-topic] {
elasticsearch {
hosts => ["https://xx:443"]
index => "logs-%{[kafka-topic]}"
action => "create"
}
}
else {
elasticsearch {
hosts => ["https://xx:443"]
index => "logs-%{[@metadata][kafka][topic]}-%{[@metadata][target]}"
action => "create"
}
}
}
}
```
and it worked well.
But when I re-create ES Cluster with same endpoint,
Logstash makes these messages.
> │ [2022-12-01T17:33:25,923][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document \_type {:es\_version=\>8} │ [2022-12-01T17:33:25,930][INFO][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data\_stream =\> auto resolved to false
> │ [2022-12-01T17:33:25,930][INFO][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data\_stream =\> auto resolved to false │ [2022-12-01T17:33:25,930][WARN][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs\_compatibility =\> v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning. │ [2022-12-01T17:33:25,930][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>
I changeed kafka input with **different group id and client id** , but those messages remain same.
Why this is happenning?
Thanks.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 1, 2022, 10:35pm UTC](https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230/2 "2022-12-01T22:35:44Z")
> [@da-head0](#):
>
> Config is not compliant with data streams
Enable --log.level debug and the [code](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/1cab1ad2a41b28ac569cb77809b48ed6b66d2b39/lib/logstash/outputs/elasticsearch/data_stream_support.rb#L152) will log why it is happening.
---
### Author:  [@da-head0](https://discuss.elastic.co/u/da-head0)
#### Post date: [December 2, 2022, 3:37am UTC](https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230/3 "2022-12-02T03:37:30Z")
> [@Badger](#):
>
> --log.level debug
Thanks for your help, Badger
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/index = "logs-%{[kafka-topic]}"
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/action = "create"
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/password =
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/id = "xx"
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/user = "xx"
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/hosts = [[https://xx:443](https://xx:443)]
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/enable\_metric = true
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/codec = \"plain\_xxx", enable\_metric=\>true, charset=\>"UTF-8"\>
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/workers = 1
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ssl\_certificate\_verification = true
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ssl\_supported\_protocols =
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/sniffing = false
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/sniffing\_delay = 5
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/timeout = 60
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/failure\_type\_logging\_whitelist =
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pool\_max = 1000
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pool\_max\_per\_route = 100
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/resurrect\_delay = 5
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/validate\_after\_inactivity = 10000
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/http\_compression = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/custom\_headers = {}
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry\_initial\_interval = 2
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry\_max\_interval = 64
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data\_stream\_type = "logs"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data\_stream\_dataset = "generic"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data\_stream\_namespace = "default"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data\_stream\_sync\_fields = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data\_stream\_auto\_routing = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/manage\_template = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/template\_overwrite = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/parent = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/join\_field = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/upsert = ""
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/doc\_as\_upsert = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script = ""
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script\_type = "inline"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script\_lang = "painless"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script\_var\_name = "event"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/scripted\_upsert = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry\_on\_conflict = 1
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pipeline = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm\_enabled = "auto"
│ [2022-12-02T12:27:20,496][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm\_pattern = "{now/d}-000001"
│ [2022-12-02T12:27:20,496][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm\_policy = "logstash-policy"
it says as these... perhaps that's because i didn't specified
data\_stream\_type
data\_stream\_dataset
data\_stream\_namespace
and set output to
```auto
index => "logs-%{[@metadata][kafka][topic]}-%{[@metadata][target]}"
action => "create"
```
for using dynamic variable for data streams...
but it worked fine before with same config.
I'm confused...
---
### Author:  [@da-head0](https://discuss.elastic.co/u/da-head0)
#### Post date: [December 12, 2022, 2:15am UTC](https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230/4 "2022-12-12T02:15:12Z")
suppose it is becuase I tried to put data stream data in data\_content node.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 9, 2023, 2:15am UTC](https://discuss.elastic.co/t/logstash-wont-send-data-to-es-after-re-create-es-cluster-kafka-input/320230/5 "2023-01-09T02:15:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.