Hello @alanstorm,
I'll start by answering your question.
The setting I am talking about is found in the APM section of Kibana. At the top you have a "settings" button you can click and then you can move to the "Indices" tab.
The last parameter is for "Metrics indices" which by default looks like this:
Now regarding your answer, I understand that metricbeat is a different feature than APM but I expected the stored documents to actually be quite similar in the end, when it comes to metrics.
To support my claim, I actually took a look at metrics data, in the apm-* index, populated by the otel-collector to see what they look like. Here is an example of some metrics about a jvm that I do see properly in the graphs on the APM page (or maybe I am completely wrong and that data is not used if the graph?). I selected a document that reports a field that is documented here: Metrics | APM Java Agent Reference [1.x] | Elastic.
{
"_index": "apm-7.15.2-metric-000001",
"_type": "_doc",
"_id": "v227bH0BroH8caugumsY",
"_version": 1,
"_score": 1,
"_source": {
"jvm.memory.heap.used": 782537896,
"jvm.memory.non_heap.used": 173870408,
"agent": {
"name": "opentelemetry/java",
"version": "1.6.0"
},
"process": {
"pid": 28559,
"command_line": "***",
"executable": "***"
},
"jvm.memory.heap.max": 1791492096,
"jvm.memory.non_heap.committed": 183042048,
"processor": {
"name": "metric",
"event": "metric"
},
"labels": {
"process_runtime_description": "Red Hat, Inc. OpenJDK 64-Bit Server VM 25.302-b08",
"deployment_name": "stage",
"service_namespace": "CEP",
"telemetry_auto_version": "1.6.0"
},
"metricset.name": "app",
"observer": {
"hostname": "2794cd8d5fad",
"name": "instance-0000000003",
"id": "debcf0c6-5cdc-47e0-a5c9-bc37927926aa",
"ephemeral_id": "97c2eab3-fd18-4ccf-b6ae-674c71f427c3",
"type": "apm-server",
"version": "7.15.2",
"version_major": 7
},
"@timestamp": "2021-11-29T17:26:19.463Z",
"ecs": {
"version": "1.11.0"
},
"service": {
"node": {
"name": "ip-172-31-26-141.ec2.internal"
},
"environment": "stage-use1",
"name": "wise-cep",
"runtime": {
"name": "OpenJDK Runtime Environment",
"version": "1.8.0_302-b08"
},
"language": {
"name": "java"
}
},
"jvm.memory.heap.committed": 1038614528,
"host": {
"hostname": "ip-172-31-26-141.ec2.internal",
"os": {
"type": "linux",
"platform": "linux",
"full": "Linux 4.14.248-129.473.amzn1.x86_64"
},
"name": "ip-172-31-26-141.ec2.internal",
"architecture": "amd64"
},
"event": {
"ingested": "2021-11-29T17:26:21.972599037Z"
}
},
"fields": {
"process.command_line.text": [
"***"
],
"jvm.memory.heap.used": [
782537920
],
"host.os.full.text": [
"Linux 4.14.248-129.473.amzn1.x86_64"
],
"labels.process_runtime_description": [
"Red Hat, Inc. OpenJDK 64-Bit Server VM 25.302-b08"
],
"jvm.memory.non_heap.committed": [
183042048
],
"observer.name": [
"instance-0000000003"
],
"host.os.full": [
"Linux 4.14.248-129.473.amzn1.x86_64"
],
"service.node.name": [
"ip-172-31-26-141.ec2.internal"
],
"host.hostname": [
"ip-172-31-26-141.ec2.internal"
],
"process.pid": [
28559
],
"service.language.name": [
"java"
],
"labels.telemetry_auto_version": [
"1.6.0"
],
"process.executable.text": [
"***"
],
"processor.event": [
"metric"
],
"jvm.memory.heap.committed": [
1038614530
],
"agent.name": [
"opentelemetry/java"
],
"host.name": [
"ip-172-31-26-141.ec2.internal"
],
"process.executable": [
"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.67.amzn1.x86_64/jre:bin:java"
],
"service.environment": [
"stage-use1"
],
"jvm.memory.non_heap.used": [
173870400
],
"jvm.memory.heap.max": [
1791492100
],
"host.os.type": [
"linux"
],
"service.name": [
"wise-cep"
],
"service.runtime.name": [
"OpenJDK Runtime Environment"
],
"labels.deployment_name": [
"stage"
],
"processor.name": [
"metric"
],
"service.runtime.version": [
"1.8.0_302-b08"
],
"observer.version_major": [
7
],
"observer.hostname": [
"2794cd8d5fad"
],
"host.architecture": [
"amd64"
],
"metricset.name": [
"app"
],
"observer.id": [
"debcf0c6-5cdc-47e0-a5c9-bc37927926aa"
],
"event.ingested": [
"2021-11-29T17:26:21.972Z"
],
"@timestamp": [
"2021-11-29T17:26:19.463Z"
],
"observer.ephemeral_id": [
"97c2eab3-fd18-4ccf-b6ae-674c71f427c3"
],
"observer.version": [
"7.15.2"
],
"host.os.platform": [
"linux"
],
"ecs.version": [
"1.11.0"
],
"observer.type": [
"apm-server"
],
"process.command_line": [
"***"
],
"agent.version": [
"1.6.0"
],
"labels.service_namespace": [
"CEP"
]
}
}
This makes me think that there isn't anything inherently different between metrics produced by otel, apm or metricbeat. However, I suspect that the APM dashboards expect specific fields for specific types of systems. That's why I was wondering if this is documented anywhere, since I don't think I can see the definition of the dashboards that are on the APM page.
Another solution would be to "reverse engineer" this, by instrumenting a node js application using the APM agent from elastic and see what the metrics documents look like. But I would rather read some documentation instead.
Once I know what the dashboards expect, I am confident I could generate the right type of document with metricbeat.
Does that make sense?
