Hi Christoffer,
thank you for sharing your observability journey and the questions you have.
When using the EDOT Java agent or OTel vanilla you should ingest the data through an EDOT/OTel collector and/or Elastic Cloud Managed OTLP endpoint (reference architecture). The details depend on your environment. This ingestion path stores the telemetry data in Elasticsearch in the OTel-native format.
- Is it currently possible to use default OpenTelemetry tools end-to-end together with the Elastic Observability solution?
You can use OTel vanilla SDKs/agents and an OTel collector to collect and ingest telemetry data. If you don't have the Elastic Cloud Managed OTLP endpoint on your ingest path you should tune the OTel contrib collector to have the right processing of data.
- If we move to the OpenTelemetry integration / EDOT OTEL Collector, are we finally on the “recommended” path, or are we likely to discover yet another better approach and end up rebuilding again?
If you use an OTel-native ingestion path, that stores the telemetry in the OTel-native format in Elasticsearch, you should be on the right path. The APM server should not be part of this ingestion path because this would result in the telemetry data being stored in ECS format.
- Will using the OpenTelemetry integration / EDOT OTEL Collector allow us to use standard OTEL tools without
labels.xxx, retain all metric attributes, and still have Kibana Observability fully working?
If you use an OTel-native ingest path, you will have OTel's span, metrics and resource attributes. We have the Elastic's OTel demo environment available. It runs the OTel demo with a mix of EDOTs and OTel vanilla SDKs/agents. Maybe have a look and explore a bit. I recommend to use Discover to explore trace and metrics documents of the ingested data.
What kind of deployment of the Elastic Stack are you using and what version?