At the time we had a work around and the problem wasn't that large but as we developed further, this has only grown in a bigger and bigger issue. Essentially the issue is that there is no legitimate link between Spans, Metadata, Metrics, Transactions, or Errors and therefore APM visualization in Kibana is completely useless.
Lets take this example. I want to know which customer experienced an error in my calculator app. To do this I create a label in each span that is "CustomerName". In my visualization I filter by the CustomerName = Matt's Car Shop. This will give me all the spans that had the customer name Matt's Car Shop. Now I want to get a count of all the errors in each spans associated transaction.
Kibana however has no way of linking the span and the transaction. This means I can't say "how many errors did Matt's Car Shop get". The only way to do this would be to search for all spans with the customer name "Matt's Car Shop", record all the transaction IDs of each span, then filter for exceptions with a particular transaction ID, for every ID I found. This means I can't make appropriate graphs or charts.
The other option would be to add a label to the metadatas context but again, no way to link the metadata back to the transaction and then filter for exceptions by those IDs, in one search.
My use case seems like something everyone would want to be able to know. How many errors did a particular customer receive in a given time period. I would like to be able to create a bar chart that shows each customer and how many errors they received but there is no option to do this.
Can anyone help? Any work arounds that people are aware of? The answer in the original post was that I could create a new index but I don't know what this means.
Lets take this example. I want to know which customer experienced an error in my calculator app. To do this I create a label in each span that is "CustomerName". In my visualization I filter by the CustomerName = Matt's Car Shop. This will give me all the spans that had the customer name Matt's Car Shop. Now I want to get a count of all the errors in each spans associated transaction.
To do this you would need to label the errors with the customer name.
If I understand the issue you describe in the original post correctly, you want to join across data types. Elasticsearch has limited support for joins, but they are not supported by APM data.
Can anyone help? Any work arounds that people are aware of?
I'm not sure if this helps or not, but the workaround is essentially just what I said above: label all the documents with the properties by which you wish to search/aggregate. If this is difficult to achieve, it would be good to dig into why - maybe there's something we can do to ease the process?
After posting this I kind of came around to understanding the backend of what apm does, a little better. As you stated, what I am trying to do is basically a join. Since this isn't a supported functionality I am limited to duplicated data injection. I am satisfied with this answer. I think my frustration stemmed from a lack of understanding. Considering the data is visible from the apm console.
Just out of curiosity. Does the apm console do a "join" in the background essentially by linking the IDs together. It seems like the console is able to make the connection between all of the IDs and how they relate to each other, so I'm assuming the backend processing is able to make this happen.
Do you mean in order to create the timeline (AKA waterfall) visualisation?
For this we first obtain a sample transaction, and then make another search against both the transaction and span indices (i.e. something like "apm-*-transaction-*,apm-*-span-*"), filtering on trace.id: <trace-id-of-transaction>. This will return all transactions and spans involved in that trace.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.