Im trying to understand and learn Elastic while working though and improving some bad setups of my predecessors. I was hoping you could help with confirming my understanding after what I have been reading.
Currently, in my logstash .conf file (one of several) I am collecting .json using a translate filter to "enrich" the data then push it to elastic. The line in my config that does this is
index => where is the actual index name.
When I check in kabana, I can easily find that index with my mappings and all. is present and been collecting data for the past 3 years. This current config is successful as it is.
Im still confused on how this works. Is logstash writing directly to the index? I dont see anything that seems related in either of the pipeline tabs (logstash pipeline or ingest pipeline)
I want to change this. I want to set it up to use ILM. I think what I will need to do is create an index template. This will allow me to define an alias and an ILM to apply to each new index. The ILM will allow me to do some purging of this data as well as keep my index sizes smaller and rotating though my warm and then cold storage.
so, my question. I think what I need to do is, create the Index template. In logstash where I point to the index, I would provide it with the index alias. This would allow me to more effectively manage ILM as well as index rollover. Correct? Should I actually have something configured in the logstash pipeline tab? as said above, I dont see anything.
again please bear in mind im pretty new to this and learning a LOT of things all at once. so please keep that in mind when trying to explain how this pipeline works