Logstash Pipeline not eligible for data streams

Elastic Stack Logstash
1

Hello,

I'm using Winlogbeat and filebeat to ingest logs into ELK, the beats agents output is logstash.

I've setup according this process order:

  • Point winlogbeat to Elasticsearch
  • run setup winlogbeat.exe setup -e
  • Start winlogbeat ... observe data getting written
  • Stop winlogbeat
  • Point winlogbeat to logstash
  • Start logstash with the config below:
input {
        beats {
                port => 5044
        }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
        hosts => "elasticsearch:9200"
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        action => "create"
        pipeline => "%{[@metadata][pipeline]}"
        user => "logstash_internal"
        password => "${LOGSTASH_INTERNAL_PASSWORD}"
    }
  } else {
     elasticsearch {
        hosts => "elasticsearch:9200"
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        action => "create"
        user => "logstash_internal"
        password => "${LOGSTASH_INTERNAL_PASSWORD}"
    }
  }
}

But I have theses errors logs:

logstash       | [2024-05-14T14:12:44,480][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash       | [2024-05-14T14:12:44,487][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_internal:xxxxxx@elasticsearch:9200/]}}
logstash       | [2024-05-14T14:12:44,501][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/"}
logstash       | [2024-05-14T14:12:44,502][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.13.2) {:es_version=>8}
logstash       | [2024-05-14T14:12:44,502][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
logstash       | [2024-05-14T14:12:44,510][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"%{[@metadata][beat]}-%{[@metadata][version]}"}
logstash       | [2024-05-14T14:12:44,510][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`

If I look on Kibana in Stack Management, I can see the data stream and Index getting data incoming:

image
image738×789 52.3 KB

Should I just ignore the said error ?

logstash       | [2024-05-14T14:12:44,510][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"%{[@metadata][beat]}-%{[@metadata][version]}"}
2

There are no errors in the logs you shared, they are INFO and WARN logs, you can ignore them.

3

Alright, thank you :+1: