Hi!
I created a visualization with APM dataview, without any filter or aggregations, at this moment I have one value to count property (5598). When I put a date histogram aggregation the count total is bigger than when I haven’t (8792).
So, trying to understand the situation, I analyzed the requests.
In the result, we can see a difference between "value" of "hits" and "buckets.doc_count".
This difference happens because of the index of data streams "metrics-apm.*.1m-default".
Removing this index from request, the number of "hits" and "doc_count" is the same.
Is the presence of this data stream (metrics 1m) in the default "APM" data view expected?
Why only this data stream 1m aren’t hidden? The other similar data streams like 10m and 60m are.
Thanks!
2 Likes
Hi, APM Dataview has multiple data streams like Metrics, Traces.
The 1m, 10m and 60m data streams are pre-aggregated results over raw transaction events done by APM Server over the time period mentioned in the name. Like 10m data stream has data aggregated over 10m for a particular transaction for service transaction metrics and produces only 1 document for the whole of 10m. Similarly 60m data stream would have 1 document created by aggregating the documents for the whole hour. These are basically internal data streams we use to display various data on the UI.
By default, you will only see 1m data stream as its not hidden. This has been done intentionally due to an internal behaviour as i mentioned these streams are used internally to display data on the APM UI.
If you want to get exact count of including the hidden data streams, you must pass an extra URL parameter in your requests like /_search?expand_wildcards=all
This way your query will hit all the data streams, even the hidden ones.
Can provide more details as to what exactly you expect from the visualisation or your use case?
Hi Achyut,
Thank you for your answer and explanation about APM metrics indexes.
In my case, my problem is just the opposite. I’m trying to get the exact count without APM metrics information because I noticed when the data view has these metrics information, it multiplies the result. I used the default APM data view, that contains “metrics-apm*” indexes.
Getting around this situation, I created a data view without “metrics-apm*” indexes. This way, using this new data view instead of the APM default data view, I could see the correct number of hits in my visualization.
So, is that the correct approach, not using APM default dataview to create custom visualizations?
I see, if you need some sort of count with out the metrics data stream, then yes you can create a new Data View specifying only the indices you require.
1 Like
