Issue with Logstash, ILM and Data stream

Elastic Stack Logstash
1

Hello all,
I'm currently trying to create a data stream with a corresponding ILM policy enabled, and then have Logstash to forward logs to that data stream.

See below steps I follow:

Create Component Template

PUT _component_template/comp_header_template
{
  "template": {
    "settings": {
      "index.lifecycle.name": "pbheaderpolicy",
      "index.lifecycle.rollover_alias": "header_stream"
    }
  }
}

Create Index Template

PUT _index_template/index_header_template
{
  "index_patterns": ["header_stream*"],
  "data_stream": { },
  "composed_of": ["comp_header_template"],
  "priority": 500
}

Mind that I have a customer ILM Policy created named pbheaderpolicy which is included inside the component template.

Logstash Output

elasticsearch {
  hosts => ["https://es01:9200"]
  user => "elastic"
  password => "*****"
  ssl_enabled => true
  cacert => "/usr/share/logstash/certs/ca/ca.crt"
  data_stream => "true"
  data_stream_type => "logs"
  data_stream_dataset => "header_stream"
  data_stream_namespace => "development"
}

Logstash works without any errors, but the data stream is getting different index template and ilm policy, as per the below:

image
image399×537 30.8 KB

I have been trying multiple outputs on logstash, but apparently it doesn't seem to be working properly..

Any assistance or further guidance over this, would be highly appreciated!
Thanks in advance.

2

Hello and welcome,

It seems that you want to create a custom data stream named header_stream, the issue here is that logstash does not support natively the creation of custom data streams.

Logstash only creates datastreams that follows the default name pattern, which is something like logs-<dataset>-<namespace>, which you can confirm since the data stream logs-header_stream-development was created.

There is a workaround to be able to create custom data streams, you basically set data_stream to false, put the name of the data stream rollover alias in the index option and set the action to create.

Something like this in your case:

output {
    elasticsearch {
        hosts => ["HOSTS"]
        index => "header_stream"
        action => "create"
        http_compression => true
        data_stream => false
        manage_template => false
        ilm_enabled => false
        cacert => 'ca.crt'
        user => 'USER'
        password => 'PASSWORD'
    }
}

The options manage_template and ilm_enabled are also set to false because the ILM policy and the template already exists in Elasticsearch.

I created an issue about this last year, but still no updates: Allow the creation of custom data streams · Issue #1152 · logstash-plugins/logstash-output-elasticsearch · GitHub

1 Like
3

Thanks a lot my friend, this works fine!