No system Metrics reported by _cat/nodes

[ben75020 (Ben)]

Hello !

I'm a beginner and I might have missed something important.
Elasticsearch is not returning any system metrics when I'm invoking the following command

GET _cat/nodes?v=true

ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.18.0.3                                                       cdfhilmrstw *      7d4ef3d44bd2

The only value returned is heap.max (probably defined statically).

Installation is Docker based (official image)

    },
    "versions": [
      "8.5.1"
    ],
    "os": {
      "available_processors": 5,
      "allocated_processors": 5,
      "names": [
        {
          "name": "Linux",
          "count": 1
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Ubuntu 20.04.5 LTS",
          "count": 1
        }
      ],
      "architectures": [
        {
          "arch": "aarch64",
          "count": 1
        }
...snip...
    "jvm": {
      "max_uptime_in_millis": 225101686,
      "versions": [
        {
          "version": "19.0.1",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "19.0.1+10-21",
          "vm_vendor": "Oracle Corporation",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 1
        }
      ],

Is there anything I'm missing ?

Thank you.

[stephenb (Stephen Brown)]

Hi @ben75020 Welcome to the community...

Hmmmm

What OS are you running this on?

      "architectures": [
        {
          "arch": "aarch64",
          "count": 1
        }

What does this return

GET /_nodes

Or
GET /_nodes/jvm,os

[ben75020 (Ben)]

Hi Stephen,

Merry Christmas and thanks for replying me.

@ben75020

          "pretty_name": "Ubuntu 20.04.5 LTS",

And host Operating System is MacOS 13.1.
This is the official docker image provided by Elastic.

GET /_nodes/jvm,os

---
{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "docker-cluster",
  "nodes": {
    "oHZpa-bERDyuDOHdC8Xeog": {
      "name": "7d4ef3d44bd2",
      "transport_address": "172.18.0.3:9300",
      "host": "172.18.0.3",
      "ip": "172.18.0.3",
      "version": "8.5.1",
      "build_flavor": "default",
      "build_type": "docker",
      "build_hash": "c1310c45fc534583afe2c1c03046491efba2bba2",
      "roles": [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "attributes": {
        "ml.allocated_processors": "5",
        "ml.max_jvm_size": "4118806528",
        "ml.allocated_processors_double": "5.0",
        "xpack.installed": "true",
        "ml.machine_memory": "8232894464"
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Linux",
        "pretty_name": "Ubuntu 20.04.5 LTS",
        "arch": "aarch64",
        "version": "5.15.49-linuxkit",
        "available_processors": 5,
        "allocated_processors": 5
      },
      "jvm": {
        "pid": 67,
        "version": "19.0.1",
        "vm_name": "OpenJDK 64-Bit Server VM",
        "vm_version": "19.0.1+10-21",
        "vm_vendor": "Oracle Corporation",
        "using_bundled_jdk": true,
        "start_time_in_millis": 1671222756720,
        "mem": {
          "heap_init_in_bytes": 4118806528,
          "heap_max_in_bytes": 4118806528,
          "non_heap_init_in_bytes": 7667712,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 0
        },
        "gc_collectors": [
          "G1 Young Generation",
          "G1 Old Generation"
        ],
        "memory_pools": [
          "CodeHeap 'non-nmethods'",
          "Metaspace",
          "CodeHeap 'profiled nmethods'",
          "Compressed Class Space",
          "G1 Eden Space",
          "G1 Old Gen",
          "G1 Survivor Space",
          "CodeHeap 'non-profiled nmethods'"
        ],
        "using_compressed_ordinary_object_pointers": "true",
        "input_arguments": [
          "-Des.networkaddress.cache.ttl=60",
          "-Des.networkaddress.cache.negative.ttl=10",
          "-Djava.security.manager=allow",
          "-XX:+AlwaysPreTouch",
          "-Xss1m",
          "-Djava.awt.headless=true",
          "-Dfile.encoding=UTF-8",
          "-Djna.nosys=true",
          "-XX:-OmitStackTraceInFastThrow",
          "-Dio.netty.noUnsafe=true",
          "-Dio.netty.noKeySetOptimization=true",
          "-Dio.netty.recycler.maxCapacityPerThread=0",
          "-Dlog4j.shutdownHookEnabled=false",
          "-Dlog4j2.disable.jmx=true",
          "-Dlog4j2.formatMsgNoLookups=true",
          "-Djava.locale.providers=SPI,COMPAT",
          "--add-opens=java.base/java.io=ALL-UNNAMED",
          "-Des.cgroups.hierarchy.override=/",
          "-XX:+UseG1GC",
          "-Djava.io.tmpdir=/tmp/elasticsearch-11083613871739996091",
          "-XX:+HeapDumpOnOutOfMemoryError",
          "-XX:+ExitOnOutOfMemoryError",
          "-XX:HeapDumpPath=data",
          "-XX:ErrorFile=logs/hs_err_pid%p.log",
          "-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m",
          "-Xms3925m",
          "-Xmx3925m",
          "-XX:MaxDirectMemorySize=2058354688",
          "-XX:G1HeapRegionSize=4m",
          "-XX:InitiatingHeapOccupancyPercent=30",
          "-XX:G1ReservePercent=15",
          "-Des.distribution.type=docker",
          "--module-path=/usr/share/elasticsearch/lib",
          "--add-modules=jdk.net",
          "-Djdk.module.main=org.elasticsearch.server"
        ]
      }
    }
  }
}

[stephenb (Stephen Brown)]

Yes I understand it's Mac OS and official docker image

What I was asking if is this the new M1/M2 ARM chip?

My Mac is intel-based Is your Mac the new M1/M2 ARM Chip?

If so, perhaps it is a bug .. or not... not sure... and I could not test that any ways.

Also do you have your docker desktop up to date?

Did you run this?

GET /_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r

And What curious exactly are you trying to solve... those values are point in timeso not really good for anything other than a quick look, if you want to monitor elasticsearch there are other approaches

[ben75020 (Ben)]

What I was asking if is this the new M1/M2 ARM chip?

Sorry, I did not understood the question at first round This is a Mac-M1.

Docker Engine was 4.13.1, I upgraded it to 4.15.0 (latest available), with no improvements.

name         du dt dup hp hc rm rp r
7d4ef3d44bd2                       cdfhilmrstw

However, top is able to retrieve system metrics.

Tasks:   6 total,   1 running,   5 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.9 us,  0.5 sy,  0.0 ni, 96.5 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :   7851.5 total,   2274.5 free,   5066.3 used,    510.7 buff/cache
MiB Swap:   1024.0 total,    833.7 free,    190.3 used.   2448.5 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                               67 elastic+  20   0 8543696   4.3g  30368 S  16.9  56.6   0:43.09 java
    1 elastic+  20   0    1936    424    360 S   0.0   0.0   0:00.01 tini                                                                   7 elastic+  20   0 2680340 101304  25668 S   0.0   1.3   0:01.78 java
   92 elastic+  20   0  103480   6116   5608 S   0.0   0.1   0:00.00 controller                                                           209 root      20   0    3884   2936   2536 S   0.0   0.0   0:00.01 bash
  217 root      20   0    5716   2588   2180 R   0.0   0.0   0:00.02 top

And What curious exactly are you trying to solve... those values are point in timeso not really good for anything other than a quick look, if you want to monitor elasticsearch there are other approaches

I'm learning elasticsearch (using the great book "elasticsearch in action", at manning publishing), and leading a study group at work. I know that they are best approaches (we are using datadog collector, in example, at work), but I'm testing several things offered into the documentation.

My message was also to ensure I did not miss a critical software component installation while setting up my lab cluster.

Ultimately, what I was trying to achieve, was getting an activity indicator per role (my current container has all possible roles), in order to identify if (and when !) it's required to dedicate some additional nodes for specific task (in example : ingestion).

[stephenb (Stephen Brown)]

@ben75020

I'm learning elasticsearch (using the great book "elasticsearch in action", at manning publishing), and leading a study group at work. I know that they are best approaches (we are using datadog collector, in example, at work), but I'm testing several things offered into the documentation.

Very cool you are getting to know / learning elastic.

Looks like a great book to get started but be careful, it is Published July 2021 which means the actual information is nearly 2 years old there is HUGE HUGE changes since then and Elastic 8.X (we are currently 8.5.3 and soon to be 8.6) is very different than 7.x

I would highly recommend augmenting your reading with our official documentation / blogs / free training etc.

@ben75020

My message was also to ensure I did not miss a critical software component installation while setting up my lab cluster.

Its Possible there is a bug... will need to check into that ...

Also to get some feature monitoring to work you may need to specify the resource limits in your docker compose.

    deploy:    
      resources:
          limits:
            cpus: '0.50'
          reservations:
            cpus: '0.25'

@ben75020

However, top is able to retrieve system metrics.

@ben75020

I know that they are best approaches (we are using datadog collector, in example, at work), but I'm testing several things offered into the documentation.

Whoa No need to use Data Dog, Elastic is a 1st Class Observability Platform this has evolved greatly in the last 2 years! We are a recognized leader in AIOps etc.

Metricbeat will collect 100s of metrics for both the host system and docker containers and elasticsearch

To get System, Docker and Elasticsearch Monitoring just enable the modules ...
Download Metricbeat and untar it on your mac.

Assuming you just used all the defaults

Enable the modules (System is enabled by default)

./metricbeat modules enable docker elasticsearch-xpack kibana-xpack

then setup

./metricbeat setup -e

The start metricbeat

./metricbeat -e

And you will get Host, Docker and Elasticsearch will show up under Stack Monitoring. See Below for images.

We also have the new Elastic Agent ... which is 1 Agent for Observability, Security and Even free endpoint protection.

Enjoy, Come back with more questions..

Built In Monitoring after I enabled Limits

Overview

Node Details..

Simple Container Dashboard (note there is a big in the I/O viz)

Host Metrics Dashboard

[ben75020 (Ben)]

Thanks Stephen for your exhaustive answer. If I can help with anything about bug reproduction/opening, let me know.

The book I'm following is the second edition, not yet published (this is the "MEAP (Meaning Early Access Program)". It covers some (may be not latest) ES 8 features.

I'll give metric beat a try very soon.

@stephenb

Enjoy, Come back with more questions..

I do have one in mind, I'll open another thread later to present it.

Thank you