I am working on the development of a monitoring application for a large scale distributed system and lately I am facing an elastic setup problem, hope you guys can help me out.
First of all, I have to point out that I have read all the available official documentations and blog posts in the matter. My experience is that all the 'hello world' kind of example described in these articles are easily applicable until you do not need to combine them. Moreover, the 'send manually request to a REST endpoint' approach is just simply not viable on a real enterprise system. I do not expect an out-of-box solution for my case, but I am just hoping that there is a thinking about automation and documenting good practices at Elastic (maybe already there are some useful guide and I just missed them).
That sad, here is my setup: I am using elastic version 7.2, all my components of the are deployed by helm into kubernetes pods, metricbeat, logstash and apm are sending data to elasticsearch (e.g. in example_data-2019.08.06-1 format) and I'd like to use custom ILM policies for optimizing resource usage.
The first problem comes when I initialize the whole setup (or redeploy elasticsearch), and beats are sent to the elastic node before I could 'install' the ILM policies with index rollover config via templates. Because of that, my custom configurations are not applied on the first set of indices.
Deleting the faulty indices manually solves the problem as the templates are applied at index creation time, new indices will have the ILM policy. The next issue is the chaos of aliases (and the rather confusing documentation). By the templates for each index pattern I am able to set the rollover_alias and write index/alias, but they are applied to every new index, so there would be multiple write index and the same rollover_alias would point to different indices. Is there an automated solution for configuring aliases and write_index?
Then when I manually set up ilm policies, templates and first indexes with aliases, I am facing the problem that rollover creates a new index, but it is empty and the docs are still written into the initial index (my guessing this is related to write_index still pointing to the original index). I believe I could achieve most of the possible IllegalArgumentExceptions (alias can point to multiple indexes, no write index, etc.), but I've never seen a stable setup.
What I am looking for is a well described practice and ideas how to configure a stable system with the mentioned components and without running curl commands manually.
Thanks in advance!