The list of transactions no longer appears after a few seconds

[btsinfo]

hello, i used 8.2.3, i use observability .
In my case, I'm using pipelines to customize transaction.name. since the upgrade to 8.3, the list of service.name transactions is displayed correctly at first, then after a few seconds no transactions are displayed.

as soon as I delete _data_stream/metrics-apm.internal-default , everything returns to normal and I manage to see all the transactions for a few seconds then the problem returns

first time ( range Today)

after fiew seconds (Range Today)

if i delete datastream it ok for few seconds

please, I performed the upgrade directly in production and my whole platform is blocked

[btsinfo]

you can reproduce the problem in any platform even laptop.

• install elasticsearch, kibana, apm-server 8.3 (apm-server legacy)

• activate instrumentation in apm-server

• in trace pipeline ( traces-apm-8.3.0
  ) add processor ( set transaction.name to {{{transaction.name}}}-{{{transaction.id}}}

• delete all datastream apm
  delete _data_stream/traces-apm-default
  delete _data_stream/logs-apm.error-default
  delete _data_stream/metrics-apm.internal-default
  delete _data_stream/metrics-apm.profiling-default

GO to kibana --> observability--apm--overview

look and you will see that the transactions appear with the new config and then no customization is displayed

• in this case do a delete datastream delete _data_stream/metrics-apm.internal-default
  look in the APM that the information is displayed correctly then after a moment we go back to square one

[lahsivjar (Vishal Raj)]

Hello @btsinfo,

I suspect the behaviour you are observing to be coming from the data Kibana uses to show transactions. Kibana prefers using aggregated metrics document with fallback to individual transaction documents for showing the transactions. The aggregated metrics documents are created by APM-Server (ref). These metrics documents use the raw transaction and not the one processed by the ingest pipeline.

So, based on above facts, once APM server produces the aggregated metrics document the UI updates to prioritise these documents over individual transaction documents. These documents are stored in metrics-apm.internal-default datastream, thus when you delete this datastream the UI shifts back to using individual transaction documents but after APM-Server again publishes a new aggregated metric document the UI is updated.

I would advise you to reconsider using transaction-id to generate transaction name. The ideal usage would be to have generic transaction names originating from your instrumentation, for example GET /query/{id} (NOTE that instead of using different IDs the name uses a generic placeholder {id} to have efficient transaction groupings).

[btsinfo]

Hello @lahsivjar
thank you for your feedback, indeed I was able to understand that. for information the use of transaction.id is just an example to reproduce the problem very quickly, in my case I am dealing with a mono servlet application and I need the other elements for an append transaction.name. this customization has great value for my team.

I was able to solve the problem by disabling the use of aggregate metrics.

I believe that transaction.name must be common in all processors (metric, span, errors, transactions) and also in terms of storage this field must not be duplicated. it's just an opinion.
Unless I am mistaken, there is default.pipeline and final.pipeline in index template and the indexes via datastream inherit this definition, in addition the two pipelines are used by the ELK eco-system and which may change in future versions, for me it will be oppotuin to have a 3 pipeline to add at the level of the custom component like custom.pipeline and which will be used to add any customization. this option will reduce the impact of upgrades to zero.

[lahsivjar (Vishal Raj)]

@btsinfo A feature to support optional and custom ingest pipelines is in the works. This issue has some details on the implementation.

[btsinfo]

@lahsivjar

thank you very much, it's a very good feature, I'll implement it soon