Good morning. I'm on a project using OpenTelementry to collect system health and status metrics for servers, and we are integrated with Elasticsearch through APM. There is a field we are setting in the data which I need to have exist at the top level of a document, but it's currently being put into labels.* namespace. I didn't think this would be an issue; should be easy enough to just add a runtime field to the metrics-apm.app index template. Caused the template to no longer work and I had to rebuild the cluster to recover. Then I realized, I could just do it via an ingest pipeline; but I didn't want to break the server again. So this time I created a clone of the metrics-apm.app-8.12.1 pipeline and added the rename processor. Then cloned the metrics-apm.app@package component template and changed the default_pipeline to point to the copy pipeline. Then I copied the metrics-apm.app index template and set it with a higher priority. Broke again. I re-tried, and didn't modify anything other than the references to point to the cloned parts. Didn't work. My question is this - is it possible to modify, or even clone and modify, managed parts of Elasticsearch? If not, why?
Some things you can change, others you cannot change as this will break the integration.
For example, the index.final_pipeline and index.default_pipeline cannot be changed, this will probably break the integration.
But you cannot for example just edit or clone a managed template or ingest pipeline, there is a series of steps that you need to follow to clone it and use as a @custom template or pipeline.
Are you using the Elastic Agent, right? You will need to create a custom ingest pipeline according to the documentation.
Every Elastic Agent integration calls a custom pipeline to allow the user to have some custom processing, but you can not edit or change the managed templates, lifecycles or ingest pipelines as this risk breaking some thing.
If you want to customize mappings or lifecycle policies as well, you will also need to follow some steps, this documentation has an example.
I'm using the otelcol-contrib (OpenTelementry collector), not an Elastic Agent. The data is coming in through APM to Elasticsearch.
Yeah, but how is the APM server running? The APM Server is now an Elastic Agent integration.
The OTel collector is an agent, and our APM server is running on a separate VM from Elastic but is integrated using the APM integration in Kibana.
The APM integration runs on an Elastic Agent, so you will need to follow the documenations if you want to change anything related to the APM integration.
I can follow the documentation up to here, but once I hit edit Elastic APM integration I don't see anywhere to edit a data stream. As far as I'm following, I went to Fleet -> Agent Policies and selected to edit Agent policy 1 -> edit Elastic APM integration. At this point I don't understand what the documentation is telling me to do because there are no data streams here. If I go back out to fleet, I can see data streams, but there is no way I can see to modify their ingest pipeline.
I do not use APM, so I'm not sure what you have.
Can you share what you have when you edit the APM integration? Share a screenshot.
The data streams mentioned in the documentation are those things like General or Real User Monitoring.
For example, in th Real User Monitoring, click on Advanced Options, it will show you more options, one of them should be related to Ingest PIpelines.
None of them have anything for ingest pipelines. However, I think I found the answer. It looks like this needs to be done in the APM Yaml file. Though, there is no 8.0 version information so this might not work. Parse data using ingest node pipelines | APM Server Reference [7.15] | Elastic
As mentioned, I do not use APM, I'm not able to see what you are seeing, what do you have in advanced options?
For example, on other integrations you have something like this:
If there is no documentation for your version, this means that is done in a different way.
APM changed on version 8.X.
There is no Ingest Pipeline or Mappings section for APM.
Not sure if it is possible to use custom mappings or ingest pipelines then.
Maybe someone from Elastic can provide more feedback.
Hi @fredyfredburger1 Welcome to the community...
If I am understanding your question...You can add a @custom pipeline, which then gets called ... you can add your field there.... You can pick your "level" of @custom
There is some docs on the stuff too here
This is ingest pipeline traces-apm-8.12.1
{
"pipeline": {
"name": "global@custom",
"ignore_missing_pipeline": true,
"description": "[Fleet] Global pipeline for all data streams"
}
},
{
"pipeline": {
"name": "traces@custom",
"ignore_missing_pipeline": true,
"description": "[Fleet] Pipeline for all data streams of type `traces`"
}
},
{
"pipeline": {
"name": "traces-apm.integration@custom",
"ignore_missing_pipeline": true,
"description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration"
}
},
{
"pipeline": {
"name": "traces-apm@custom",
"ignore_missing_pipeline": true,
"description": "[Fleet] Pipeline for the `apm` dataset"
}
}
]
Same concept for mappings
They say managed but my experience is that you can update them and they will persist.
I'm curious, shouldn't the APM integration expose this option to the user in the Fleet UI as all the other integrations?
I think some of this functionality will eventually get to the UI .. but it is starting to get in the docs 
The whole APM area is pretty complex... With the apm agent integration or self-managed APM severi and or open telemetry...
not to mention the language agents vs open telemetry agents..
There's a lot going on there
Thank you all for the help. @stephenb was correct. I tested and figured out that I can actually modify the metrics-apm.app pipeline without breaking anything. That allowed me to just insert a field rename at the bottom and it worked perfectly. Before that I tested adding pipelines to the apm.yaml file and that did not work, so I do agree with @leandrojmp that it looks like that feature probably was removed in version 8.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.