Issue with Logstash, ILM and Data stream

Elastic Stack Logstash
1

Hello all,
I'm currently trying to create a data stream with a corresponding ILM policy enabled, and then have Logstash to forward logs to that data stream.

See below steps I follow:

Create Component Template

PUT _component_template/comp_header_template
{
  "template": {
    "settings": {
      "index.lifecycle.name": "pbheaderpolicy",
      "index.lifecycle.rollover_alias": "header_stream"
    }
  }
}

Create Index Template

PUT _index_template/index_header_template
{
  "index_patterns": ["header_stream*"],
  "data_stream": { },
  "composed_of": ["comp_header_template"],
  "priority": 500
}

Mind that I have a customer ILM Policy created named pbheaderpolicy which is included inside the component template.

Logstash Output

elasticsearch {
  hosts => ["https://es01:9200"]
  user => "elastic"
  password => "*****"
  ssl_enabled => true
  cacert => "/usr/share/logstash/certs/ca/ca.crt"
  data_stream => "true"
  data_stream_type => "logs"
  data_stream_dataset => "header_stream"
  data_stream_namespace => "development"
}

Logstash works without any errors, but the data stream is getting different index template and ilm policy, as per the below:

image
image399×537 30.8 KB

I have been trying multiple outputs on logstash, but apparently it doesn't seem to be working properly..

Any assistance or further guidance over this, would be highly appreciated!
Thanks in advance.

2

Hello and welcome,

It seems that you want to create a custom data stream named header_stream, the issue here is that logstash does not support natively the creation of custom data streams.

Logstash only creates datastreams that follows the default name pattern, which is something like logs-<dataset>-<namespace>, which you can confirm since the data stream logs-header_stream-development was created.

There is a workaround to be able to create custom data streams, you basically set data_stream to false, put the name of the data stream rollover alias in the index option and set the action to create.

Something like this in your case:

output {
    elasticsearch {
        hosts => ["HOSTS"]
        index => "header_stream"
        action => "create"
        http_compression => true
        data_stream => false
        manage_template => false
        ilm_enabled => false
        cacert => 'ca.crt'
        user => 'USER'
        password => 'PASSWORD'
    }
}

The options manage_template and ilm_enabled are also set to false because the ILM policy and the template already exists in Elasticsearch.

I created an issue about this last year, but still no updates: Allow the creation of custom data streams · Issue #1152 · logstash-plugins/logstash-output-elasticsearch · GitHub

2 Likes
3

Thanks a lot my friend, this works fine!

4

I am having the same issue when I configure logstash to create the datastream, it default to use "log" template and "log" ILM.

with the suggestion you provided, I will need to create the datasream manually right? I still not sure how it will use the data_stream if data_stream is set to true in logstash's ES output?

thanks

5

No, you need to create a data stream template first, like the example in the first post.

Something like this:

PUT _index_template/your-index-template
{
  "index_patterns": ["your-data-stream-pattern"],
  "data_stream": { },
  "composed_of": ["comp_header_template"],
  "priority": 500
}

Then you use an output like the one I suggested, with data_stream => false.

If you set data_stream as true in the Logstash Elasticsearch output, your datastream name will need to be something like logs-dataset-namespace and it will use the logs template and log ILM.

As mentioned in my previous answer logstash does not support custom data streams, the only way to write to custom data streams with different name patterns is with the workaround shared.

6

@leandrojmp thanks a lot. I was able to configure the custom datastream now.