Hi All,
i am using apm server standalone to ingest traces from my application to Elasticsearch via opentelemetry collecor. i configured APM integration inside Kibana to point to my apm server and it works fine.i recently upgraded Elasticsearch, kibana, and apm to 8.12 version, old version was 8.7.0 and found out apm was trying to push traces through old traces-apm-8.7.0 ingest pipeline and failing with below error .
failed to index document in 'traces-apm.default' (fail_processor_exception): Document produced by APM Server v8.12.0, which is newer than the installed APM integration (v8.7.0-preview). The APM integration must be upgraded.
when i checked old ingest pipline was still there so i cleaned it up and reconfigured apm integration. now i get new error in apm as below
failed to index documents in 'traces-apm-default' (illegal_argument_exception): pipeline with id [traces-apm-8.7.0] does not exist
so i stopped apm server, uninstalled apm integration, manually cleaned ingest pipelines related to apm integrations if any and also cleaned all apm related indexes except below two. i could not delete these two indexes despite i am having superuser role.
.apm-agent-configuration
.apm-custom-link
then i ignored it and reconfigured apm integration and started apm server again but i still get error
failed to index documents in 'traces-apm-default' (illegal_argument_exception): pipeline with id [traces-apm-8.7.0] does not exist
I am now clueless about what went wrong in the cluster that is telling APM to ingest traces to the old pipeline which does not exist and how to fix it. I would appreciate any help fixing this issue as it as its affecting my QA user's observability capabilities in distributed tracing.