I have a pretty small amount of documents from Metricbeat and I have loaded them into 2 ways into Elasticsearch:
Using the default Metricbeat index template and datastream (created by Metricbeat), ILM policy has created 4 indexes with rollover every hour.
Writing to a regular index by Metricbeat with index name pattern that contains hour in its name, so also have 3 indexes to search into.
And what I see is that searching in Kibana using the data stream indexes is drastically slow compared to regular indexes search. I have noted that when searching in a data stream, Kibana sends 5 similar search requests with a pretty big delay between each other.
Looks like this is some bug or misconfiguration of the index template created by Metricbeat. When I have created the same index template with the data stream manually (but without default mappings) and ILM, everything works fine.
Looks like I should use the default Metricbeat index template settings and mappings, since other apps, like Observability, rely on it. Otherwise, they just don't work.
So that issue is still present.
Can someone please help?
Thanks for the detailed report. What version of Metricbeat are you using?
I wonder if this is caused by the big number of field mappings included in Metricbeat. Did you have the chance of trying with Elastic Agent? It also uses data streams, but one is created for each data set of each integration, with much smaller mappings. This would be more similar to the test you mention that works better.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
