Trying to understand a previous setup/pipeline

[jason3 (jason)]

Hello all,

Im trying to understand and learn Elastic while working though and improving some bad setups of my predecessors. I was hoping you could help with confirming my understanding after what I have been reading.

Currently, in my logstash .conf file (one of several) I am collecting .json using a translate filter to "enrich" the data then push it to elastic. The line in my config that does this is
index => where is the actual index name.

When I check in kabana, I can easily find that index with my mappings and all. is present and been collecting data for the past 3 years. This current config is successful as it is.

Im still confused on how this works. Is logstash writing directly to the index? I dont see anything that seems related in either of the pipeline tabs (logstash pipeline or ingest pipeline)

I want to change this. I want to set it up to use ILM. I think what I will need to do is create an index template. This will allow me to define an alias and an ILM to apply to each new index. The ILM will allow me to do some purging of this data as well as keep my index sizes smaller and rotating though my warm and then cold storage.

so, my question. I think what I need to do is, create the Index template. In logstash where I point to the index, I would provide it with the index alias. This would allow me to more effectively manage ILM as well as index rollover. Correct? Should I actually have something configured in the logstash pipeline tab? as said above, I dont see anything.

again please bear in mind im pretty new to this and learning a LOT of things all at once. so please keep that in mind when trying to explain how this pipeline works

[stephenb (Stephen Brown)]

Hi @jason3 Welcome to the community.

In general, it seems that you're understanding concepts quite well
You did not mention what version of the stack you are on.

Yes, you are want to create a template , an ILM policy and then a write alias for a rollover.

There are some significant differences you are using the 7.x stack for 8.x stack.

In 8.x the concepts are focused on data streams which abstract some of the configuration in 7.x

@jason3

Correct? Should I actually have something configured in the logstash pipeline tab? as said above, I dont see anything.

No, you do not need anything and either of those two tabs.

The logstash screen is for centralized log stash pipeline management, there's some minor pros and cons of using that, but today you're not using that so that's fine. Just keep doing what you're doing.

Ingest pipelines are processing pipelines that actually run inside elasticsearch. There's some pros and cons to running those. But today you're using logstash which is which is just fine ...

I would focus on what you're looking at today... getting your indices or data streams depending on which one you use under ILM which makes sense.

Make sure you are looking at the proper documentation 7.x or 8.x. there is some significance differences and if you're using 7.x and looking at the 8.x documentation you're going to get in trouble

Come back with some specific questions when you have them.