Hot-Warm Arch, without ILM

Elastic Stack Logstash
1

Long time lurker, but first time poster! So lets see if this works!

Currently implementing a Hot-Warm architecture. Which consists of
3 dedicated Master Nodes
3 Hot Nodes
3 Warm Nodes
The Attrib is set by
node.attr.box_type: master/hot/warm

Now what is not Elastic specific is that the cluster is deployed using Salt Stack, which comes into play later.

i Ingest my data from File/Audit-Beat etc. through a Logstash Pipe.
Output Looking like
elasticsearch { hosts => ["elasticsearch-hot-01:9200","elasticsearch-hot-02:9200","elasticsearch-hot-03:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" }
And that creates daily indices as intended.

Now comes the issue, which is that logstash is creating the indicies on all nodes. But i would only like it to use the Hot nodes as those are on SSDs. Makes sense right!
Now what i really dont want to do is to use the Rest API to ILM the issue. Simply because that would create a really bad implementation with the Salt state.
Moving the index to Warm nodes will be achieved using Curator.

So simply put, is there any way to using logstash to set the created indexes to only exist on Hot, instead of Hot and Warm nodes? without having to use Rest API Calls?

2

You generally control this through an index tempate, which automatically assigns the appropraite settings on index creation, resulting in all new indices being created on the hot nodes.

You can manage this yourself or create one and let Logstash upload it on startup.

3

So i assume that theres no easy way other then doing the index lifecycle management way in making logstash create indexes on the "hot" nodes?

4

I am not sure I understand your question. The index template specifies the settings that makes new indices be allocated to hot nodes. You can then use ILM or Curator to manage the lifecycle. If you are looking for some other way to manage this I am not aware of any.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.