@shaunak I would like to express my gratitude for your insightful blog post titled 'Elastic Stack monitoring with Metricbeat via Logstash or Kafka' It has been instrumental in helping me set up a centralized monitoring system. Thank you for sharing your expertise.
Currently, I am in the process of migrating to ES 8.7 while aiming to maintain my existing monitoring architecture. Unfortunately, I have encountered a challenge during this transition. Instead of solely seeking support for my specific issue, I wanted to reach out to you because I believe an updated blog post dedicated to ES 8.7 would immensely benefit many others. Such a post could offer a more comprehensive and structured approach for those seeking a centralized monitoring solution.
Thank you for taking the time to consider my request. I appreciate your valuable insights.
While we anticipate an update to the article mentioned above, I wanted to share a solution for a potential issue you might encounter. You may come across an error message stating '"reason"=>"only write ops with an op_type of create are allowed in data streams"'. After extensive research, including various blog posts and Elastic's documentation, I found that the most straightforward solution is to revert to using indices, as data streams were not designed to handle update operations.
Here's a step-by-step guide on how to do it:
- Stop any incoming monitoring data. It's crucial to do this first to prevent any data loss or corruption.
- Go to the index templates.
- Open each template that begins with '.monitoring-'.
- Please note that these are system templates, so you'll need to select 'show system templates' to view them.
- Untick the option that says "Create data stream" and then save your changes.
By following these steps, you should be able to circumvent the mentioned issue. I hope this saves you some time and effort.