Logstash won't send data to ES after re-create ES Cluster(Kafka Input)

[da-head0 (Dahye)]

Hello,

ELK version : 8.4.2
Environment: Logstash in EKS, ECK Elasticsearch

Hello, I used to send data to Elasticsearch with these config.

    input {
        kafka {
            bootstrap_servers => "xx:9094, xx2:9094"
            topics_pattern => ".*"
            consumer_threads => 4
            codec => "json"
            decorate_events => "basic"
            group_id => "kafka"
            client_id => "logstash"
            security_protocol => "SSL"
            ssl_truststore_location => "/logstash/xx"
            enable_auto_commit => "true"
            partition_assignment_strategy => "round_robin"
            auto_commit_interval_ms => 1000
        }
    output {
        if [kafka-topic] {
            elasticsearch {
                hosts => ["https://xx:443"]
                index => "logs-%{[kafka-topic]}"
                action => "create"
            }
        }
        else {
            elasticsearch {
                hosts => ["https://xx:443"]
                index => "logs-%{[@metadata][kafka][topic]}-%{[@metadata][target]}"
                action => "create"
            }
        }
    }
    }

and it worked well.

But when I re-create ES Cluster with same endpoint,

Logstash makes these messages.

│ [2022-12-01T17:33:25,923][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8} │ [2022-12-01T17:33:25,930][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
│ [2022-12-01T17:33:25,930][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false │ [2022-12-01T17:33:25,930][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs_compatibility => v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning. │ [2022-12-01T17:33:25,930][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>

I changeed kafka input with different group id and client id, but those messages remain same.

Why this is happenning?

Thanks.

[Badger]

@da-head0

Config is not compliant with data streams

Enable --log.level debug and the code will log why it is happening.

[da-head0 (Dahye)]

@Badger

--log.level debug

Thanks for your help, Badger

│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/index = "logs-%{[kafka-topic]}"
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/action = "create"
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/password =
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/id = "xx"
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/user = "xx"
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/hosts = [https://xx:443]
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/enable_metric = true
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/codec = <LogStash::Codecs::Plain id=>"plain_xxx", enable_metric=>true, charset=>"UTF-8">
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/workers = 1
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ssl_certificate_verification = true
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ssl_supported_protocols =
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/sniffing = false
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/sniffing_delay = 5
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/timeout = 60
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/failure_type_logging_whitelist =
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pool_max = 1000
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pool_max_per_route = 100
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/resurrect_delay = 5
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/validate_after_inactivity = 10000
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/http_compression = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/custom_headers = {}
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry_initial_interval = 2
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry_max_interval = 64
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_type = "logs"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_dataset = "generic"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_namespace = "default"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_sync_fields = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_auto_routing = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/manage_template = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/template_overwrite = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/parent = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/join_field = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/upsert = ""
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/doc_as_upsert = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script = ""
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script_type = "inline"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script_lang = "painless"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script_var_name = "event"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/scripted_upsert = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry_on_conflict = 1
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pipeline = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm_enabled = "auto"
│ [2022-12-02T12:27:20,496][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm_pattern = "{now/d}-000001"
│ [2022-12-02T12:27:20,496][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm_policy = "logstash-policy"

it says as these... perhaps that's because i didn't specified
data_stream_type
data_stream_dataset
data_stream_namespace

and set output to

index => "logs-%{[@metadata][kafka][topic]}-%{[@metadata][target]}"
action => "create"

for using dynamic variable for data streams...

but it worked fine before with same config.
I'm confused...

[da-head0 (Dahye)]

suppose it is becuase I tried to put data stream data in data_content node.