Logstash won't send data to ES after re-create ES Cluster(Kafka Input)

Elastic Stack Logstash
1

Hello,

ELK version : 8.4.2
Environment: Logstash in EKS, ECK Elasticsearch

Hello, I used to send data to Elasticsearch with these config.

    input {
        kafka {
            bootstrap_servers => "xx:9094, xx2:9094"
            topics_pattern => ".*"
            consumer_threads => 4
            codec => "json"
            decorate_events => "basic"
            group_id => "kafka"
            client_id => "logstash"
            security_protocol => "SSL"
            ssl_truststore_location => "/logstash/xx"
            enable_auto_commit => "true"
            partition_assignment_strategy => "round_robin"
            auto_commit_interval_ms => 1000
        }
    output {
        if [kafka-topic] {
            elasticsearch {
                hosts => ["https://xx:443"]
                index => "logs-%{[kafka-topic]}"
                action => "create"
            }
        }
        else {
            elasticsearch {
                hosts => ["https://xx:443"]
                index => "logs-%{[@metadata][kafka][topic]}-%{[@metadata][target]}"
                action => "create"
            }
        }
    }
    }

and it worked well.

But when I re-create ES Cluster with same endpoint,

Logstash makes these messages.

│ [2022-12-01T17:33:25,923][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8} │ [2022-12-01T17:33:25,930][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
│ [2022-12-01T17:33:25,930][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false │ [2022-12-01T17:33:25,930][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs_compatibility => v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning. │ [2022-12-01T17:33:25,930][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>

I changeed kafka input with different group id and client id, but those messages remain same.

Why this is happenning?

Thanks.

2

Enable --log.level debug and the code will log why it is happening.

3

Thanks for your help, Badger

│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/index = "logs-%{[kafka-topic]}"
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/action = "create"
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/password =
│ [2022-12-02T12:27:20,493][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/id = "xx"
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/user = "xx"
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/hosts = [https://xx:443]
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/enable_metric = true
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/codec = <LogStash::Codecs::Plain id=>"plain_xxx", enable_metric=>true, charset=>"UTF-8">
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/workers = 1
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ssl_certificate_verification = true
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ssl_supported_protocols =
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/sniffing = false
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/sniffing_delay = 5
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/timeout = 60
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/failure_type_logging_whitelist =
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pool_max = 1000
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pool_max_per_route = 100
│ [2022-12-02T12:27:20,494][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/resurrect_delay = 5
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/validate_after_inactivity = 10000
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/http_compression = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/custom_headers = {}
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry_initial_interval = 2
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry_max_interval = 64
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_type = "logs"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_dataset = "generic"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_namespace = "default"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_sync_fields = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/data_stream_auto_routing = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/manage_template = true
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/template_overwrite = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/parent = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/join_field = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/upsert = ""
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/doc_as_upsert = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script = ""
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script_type = "inline"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script_lang = "painless"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/script_var_name = "event"
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/scripted_upsert = false
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/retry_on_conflict = 1
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/pipeline = nil
│ [2022-12-02T12:27:20,495][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm_enabled = "auto"
│ [2022-12-02T12:27:20,496][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm_pattern = "{now/d}-000001"
│ [2022-12-02T12:27:20,496][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::Elasticsearch/ilm_policy = "logstash-policy"

it says as these... perhaps that's because i didn't specified
data_stream_type
data_stream_dataset
data_stream_namespace

and set output to

index => "logs-%{[@metadata][kafka][topic]}-%{[@metadata][target]}"
action => "create"

for using dynamic variable for data streams...

but it worked fine before with same config.
I'm confused...

4

suppose it is becuase I tried to put data stream data in data_content node.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.