How am I supposed to understand why the field is confusing?

Elastic Stack Logstash
1

Good afternoon, colleagues. I have been playing for several hours with this incomprehensible error, how can this be - someone can explain the dependencies of fields between different products, you can link to the documentation, in general the documentation is terrible - you have to look for answers to one question in the track of different links, but this is to the topic does not apply, how to understand the essence of such errors?

[ERROR][logstash.outputs.elasticsearch][main] Invalid data stream configuration, following parameters are not supported: {"index"=>"docker-000001"}
[2021-11-14T09:50:57,530][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid data stream configuration: ["index"]>

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/certs/ca/ca.crt"]
    ssl_key => '/certs/logstash/logstash.pkcs8.key'
    ssl_certificate => '/certs/logstash/logstash.crt'
    ssl_verify_mode => "force_peer"
  }
}
output {
  elasticsearch {
    hosts => ["https://es00:9200"]
    cacert => '/certs/ca/ca.crt'
    ssl => true
    user => 'logstash_internal'
    password => 'JQhhDoQP23MN2oXTjieP'
    data_stream => true
    index => "docker-000001"
  }
}


filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
output.logstash:
  hosts: ["logstash:5044"]
  ssl.certificate_authorities: ["/certs/ca/ca.crt"]
  ssl.certificate: "/certs/filebeat/filebeat.crt"
  ssl.key: "/certs/filebeat/filebeat.key"
  index: "docker-%{+yyyy.MM.dd}"
  ssl.enabled: true
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "docker-alias"
setup.ilm.pattern: "{now/d}-000001"

PUT _index_template/docker-template
{
  "index_patterns": ["docker*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "docker-ilm",      
      "index.lifecycle.rollover_alias": "docker-alias"    
    }
  }
}
PUT docker-000001
{
  "aliases": {
    "docker-alias": {
      "is_write_index": true,
  "settings": {
      "number_of_shards": 1,  
      "number_of_replicas": 1
    }
  }
}

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "docker-000001",
        "alias": "docker-alias",
        "is_write_index": "true"
      }
    }
  ]
}
2

When you use a data stream the index names are auto-generated, so you cannot use the index option on the output to set the name.

1 Like
3

I may have a difficulty with the logic - help with a resource for learning?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.