Apm system.memory.actual.free and system.memory.total not exists in mapping

Elastic Observability
1

Kibana version :
Kibana version 8.1.2

Elasticsearch version :
Elasticsearch version 8.1.2

APM Server version :
fleet integrated 8.1.2

Original install method (e.g. download page, yum, deb, from source, etc.) and version :
download page

Fresh install or upgraded from other version?
Fresh

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant) :

I installed ElasticSrarch and Kibana after that added integrated fleet and Apm on windows server 2019. For client I use AspNetCore sample form OpenTelemetry Github repository. When I send to and try access apm services from observability I get below error.

KibanaError
KibanaError1611×781 150 KB

And this error logged:

[2022-04-18T14:34:22.619+04:30][ERROR][plugins.apm] Error: search_phase_execution_exception: [script_exception] Reason: link error
    at D:\ElasticStack\kibana\x-pack\plugins\observability\common\utils\unwrap_es_response.js:60:11
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at getServiceInstancesSystemMetricStatistics (D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\services\get_service_instances\get_service_instances_system_metric_statistics.js:126:20)
    at async Promise.all (index 1)
    at D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\services\get_service_instances\main_statistics.js:28:51
    at async Promise.all (index 0)
    at handler (D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\services\route.js:622:45)
    at D:\ElasticStack\kibana\x-pack\plugins\apm\server\routes\apm_routes\register_apm_server_routes.js:143:13
    at Router.handle (D:\ElasticStack\kibana\src\core\server\http\router\router.js:163:30)
    at handler (D:\ElasticStack\kibana\src\core\server\http\router\router.js:124:50)
    at exports.Manager.execute (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\toolkit.js:60:28)
    at Object.internals.handler (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\handler.js:46:20)
    at exports.execute (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\handler.js:31:20)
    at Request._lifecycle (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\request.js:371:32)
    at Request._execute (D:\ElasticStack\kibana\node_modules\@hapi\hapi\lib\request.js:281:9)
Response: {
  error: {
    root_cause: [
      {
        type: 'script_exception',
        reason: 'link error',
        script_stack: [ "doc['system.memory.actual.free']", '     ^---- HERE' ],
        script: "1 - doc['system.memory.actual.free'] / doc['system.memory.total']",
        lang: 'expression'
      }
    ],
    type: 'search_phase_execution_exception',
    reason: 'all shards failed',
    phase: 'query',
    grouped: true,
    failed_shards: [
      {
        shard: 0,
        index: '.ds-metrics-apm.internal-default-2022.04.17-000001',
        node: 'KsqpLySITbOWAlCPlvsdSg',
        reason: {
          type: 'script_exception',
          reason: 'link error',
          script_stack: [ "doc['system.memory.actual.free']", '     ^---- HERE' ],
          script: "1 - doc['system.memory.actual.free'] / doc['system.memory.total']",
          lang: 'expression',
          caused_by: {
            type: 'parse_exception',
            reason: 'Field [system.memory.actual.free] does not exist in mappings'
          }
        }
      }
    ]
  },
  status: 400
}

Also in discovery metric-* data view I have system.memory.actual.free and system.memory.total (some of them have value and some of them don't have).
Why this message and error appears and how can I solve it?

2 Likes
2

I have same problem but I don't know how to resolve it :neutral_face:

3

I've also encountered this issue, does anyone have a solution?

4

I was able to get rid of the error message (I'm still waiting for data to show up), I'm not sure if this fixes the problem but at least you'll get rid of the error message.
Go to "Index Management" -> Search for the index with issues (from the log above it seems it's ".ds-metrics-apm.internal-default-2022.04.17-000001") -> Select the index -> Go to "Edit Settings" add the following line "system.memory.actual.free", anywhere in the "index.query.default_field" section.
image

1 Like
5

We also get the same error, but adding the field in the settings doesn't cut it for us. Any news about this issue?

6

Sorry for being late. Two weeks ago I found the problem, and I think It's a bug.
You see the error If you don't have any CPU and Memory metrics at your Elastic Metrics index. In my case, I use the OpenTelemetry library at the .Net Core project, It didn't send CPU and Memory benchmarks, so I got the error. The error disappeared when I added the Elastic APM library to my project and sent single CPU and memory metrics.