Metricbeat is not capturing CPU and DISK information

chandukreddi · October 18, 2018, 9:30pm

Hello Team,

I have enabled system module in metricbeat but all I see is Processora details in Elasticsearch but I don;t find anything CPU/CORE/DISK related information captured. any idea?

system.yml
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/modules.d/system.yml

- module: system

  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - core
    - diskio
    #- socket
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime


**metricbeat.yml**
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/metricbeat.yml | grep setup
setup.template.name: "cassandra-beats"
setup.template.pattern: "cas-beats-*"
setup.template.settings:
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
#setup.dashboards.url:
setup.kibana:
# `setup.kibana.host` options.


Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/metricbeat.yml | grep elastic -A10

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.1.27.6:9200"]
  index: "logs_write_cas_beat"
  #template: "/etc/metricbeat/cass_beat_2.json"
  #template_name: "cassandra_beat"
  #template_overwrite: true

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

Thanks
Chandra

Mario_Castro · October 20, 2018, 5:59pm

Hi @chandukreddi

Can you specify your metricbeat version and you OS? I guess is some kind of Linux

Best regards

chandukreddi · October 26, 2018, 6:14pm

Sorry Mario for delay.

beats: metricbeat-6.1.3-x86_64.rpm
OS: Red Hat Enterprise Linux Server release 7.4

Thanks
Chandra

chandukreddi · October 29, 2018, 3:01am

Hello Experts,

Can anyone help me? I am new to metricsbeat and I stuck.

Thanks
Chandra

Andrew_Cholakian1 · October 29, 2018, 7:13pm

@chandukreddi can you share one of the Elasticsearch documents that are being created in metricbeat? It'd be useful to see exactly which fields are being populated.

chandukreddi · November 5, 2018, 11:01pm

Hi Andrew,

Sorry, some reason I am not getting alerts when topic get updated.

Here you go!

{
"_index": "cas-beat-2018.10.17-000002",
"_type": "doc",
"_id": "vS-gy2YBhjpsdWfhvk22",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-10-31T20:20:45.784Z",
"system": {
"process": {
"pgid": 74076,
"cmdline": "python /usr/local/dse-5.0.14/resources/cassandra/bin/cqlsh.py",
"cwd": "/root",
"fd": {
"open": 8,
"limit": {
"soft": 120000,
"hard": 120000
}
},
"state": "sleeping",
"username": "root",
"cpu": {
"total": {
"pct": 0.004,
"norm": {
"pct": 0.0001
}
},
"start_time": "2018-10-31T19:49:22.000Z"
},
"ppid": 70795,
"name": "python",
"memory": {
"size": 550658048,
"rss": {
"bytes": 35090432,
"pct": 0.0003
},
"share": 4681728
},
"pid": 74076
}
},
"metricset": {
"module": "system",
"name": "process",
"rtt": 83020
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
},
"fields": {
"@timestamp": [
"2018-10-31T20:20:45.784Z"
],
"system.process.cpu.start_time": [
"2018-10-31T19:49:22.000Z"
]
},
"sort": [
1541017245784
]
}

chandukreddi · November 7, 2018, 10:40pm

@Andrew_Cholakian1, Could you please help me?

Thanks
Chandra

Andrew_Cholakian1 · November 7, 2018, 11:43pm

So, the filesystem metricset in your config is configured to only send events once per minute. That means only some events will have filesystem data.

You can read more about that here.

In the document you shared I do see CPU stats, so that seems to be working correctly.

chandukreddi · November 8, 2018, 7:22pm

Thanks Andrew!

In case your referring to below lines, I believe that is related to that particular process right?

I am looking for overall host CPU usage and LOAD, on top that it also missing diskio, if you look at overall document it just captured only system module process info.

"system": {
"process": {

process:
"pgid": 74076,
"cmdline": "python /usr/local/dse-5.0.14/resources/cassandra/bin/cqlsh.py",

"cpu": {
"total": {
"pct": 0.004,
"norm": {
"pct": 0.0001
}
},
"start_time": "2018-10-31T19:49:22.000Z"
},

Andrew_Cholakian1 · November 8, 2018, 8:26pm

It looks like you're on windows, but windows doesn't offer a load statistic.

chandukreddi · November 8, 2018, 10:09pm

Hi Andrew,

its not windows, its Linux.

Thanks
Chandra

Andrew_Cholakian1 · November 9, 2018, 12:33am

Do you have the system load metricset enabled? https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-load.html

chandukreddi · November 9, 2018, 5:53pm

Hi @Andrew_Cholakian1,

Yes I have enabled on sytem.yml file, Do I need to enable anywhere else? and I do see those fields in my Kibana when I disable option hide missing fields, this means its enabled but not capturing any data.

--- System module parameters

module: system

period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
- core
- diskio

chandukreddi · November 19, 2018, 4:33pm

@all , Can anyone help me to understand why I can't see Load/CPU/DISK IO info in elasticsearch docs?

Thanks
Chandra

Andrew_Cholakian1 · November 19, 2018, 4:39pm

Hmmm, it should be showing up. I'm not sure why it isn't. @ruflin or @shaunak do you have any ideas here?

shaunak · November 19, 2018, 4:57pm

Yes, it's odd that it's not showing up.

@chandukreddi Could you try the following (temporarily for debugging purposes only) to help and narrow down the issue?

Disable all modules except system.
Edit your ./modules.d/system.yml and only enable the cpu metricset.
Start up Metricbeat but with the -e -d"*" flags. This will generate debugging output. Could you please post this here? Just the first 10 seconds or so of output should be sufficient.

Thanks,

Shaunak

chandukreddi · November 19, 2018, 5:07pm

Okay @shaunak let me try that.

Thanks
Chandra

chandukreddi · November 19, 2018, 5:23pm

Hi @shaunak
2018/11/19 22:47:16.180010 wrapper.go:149: DBG [module] Starting metricSetWrapper[module=system, name=diskio, host=]
2018/11/19 22:47:16.350588 processor.go:275: DBG [publish] Publish event: {
"@timestamp": "2018-11-19T22:47:16.180Z",
"@metadata": {
"beat": "metricbeat",
"type": "doc",
"version": "6.1.3"
},
"system": {
"diskio": {
"name": "sdg",
"read": {
"count": 137,
"time": 20,
"bytes": 2334720
},
"write": {
"bytes": 0,
"count": 0,
"time": 0
},
"io": {
"time": 17
},
"iostat": {
"service_time": 0,
"busy": 0,
"read": {
"request": {
"merges_per_sec": 0,
"per_sec": 0
},
"per_sec": {
"bytes": 0
}
},
"write": {
"request": {
"merges_per_sec": 0,
"per_sec": 0
},
"per_sec": {
"bytes": 0
}
},
"queue": {
"avg_size": 0
},
"request": {
"avg_size": 0
},
"await": 0
},
"serial_number": "36d094660757264002333866362b40e36"
}
},
"metricset": {
"module": "system",
"name": "diskio",
"rtt": 170174
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
}
Thanks
Chandra

chandukreddi · November 19, 2018, 5:24pm

My System module.
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/modules.d/system.yml

module: system

period: 10s
metricsets:
- cpu

- load

- memory

- network

- process

- process_summary

- core

- diskio

#- socket

processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory

module: system
period: 1m
metricsets:
- filesystem
- fsstat
  processors:
- drop_event.when.regexp:
  system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
module: system
period: 15m
metricsets:
- uptime

Thanks
Chandra

chandukreddi · November 19, 2018, 5:32pm

Here is the CPU module stats

2018/11/19 22:47:15.825514 wrapper.go:149: DBG [module] Starting metricSetWrapper[module=system, name=cpu, host=]
2018/11/19 22:47:15.828325 processor.go:275: DBG [publish] Publish event: {
"@timestamp": "2018-11-19T22:47:15.825Z",
"@metadata": {
"beat": "metricbeat",
"type": "doc",
"version": "6.1.3"
},
"system": {
"cpu": {
"irq": {
"pct": 0
},
"softirq": {
"pct": 0
},
"user": {
"pct": 0
},
"idle": {
"pct": 0
},
"steal": {
"pct": 0
},
"system": {
"pct": 0
},
"iowait": {
"pct": 0
},
"total": {
"pct": 0
},
"nice": {
"pct": 0
},
"cores": 32
}
},
"metricset": {
"name": "cpu",
"rtt": 2472,
"module": "system"
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
}

Thanks
Chandra