Metricbeat is not capturing CPU and DISK information


(Chandukreddi) #1

Hello Team,

I have enabled system module in metricbeat but all I see is Processora details in Elasticsearch but I don;t find anything CPU/CORE/DISK related information captured. any idea?

system.yml
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/modules.d/system.yml

- module: system

  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - core
    - diskio
    #- socket
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime


**metricbeat.yml**
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/metricbeat.yml | grep setup
setup.template.name: "cassandra-beats"
setup.template.pattern: "cas-beats-*"
setup.template.settings:
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
#setup.dashboards.url:
setup.kibana:
# `setup.kibana.host` options.


Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/metricbeat.yml | grep elastic -A10

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.1.27.6:9200"]
  index: "logs_write_cas_beat"
  #template: "/etc/metricbeat/cass_beat_2.json"
  #template_name: "cassandra_beat"
  #template_overwrite: true

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

Thanks
Chandra


(Mario Castro) #2

Hi @chandukreddi :slight_smile:

Can you specify your metricbeat version and you OS? I guess is some kind of Linux

Best regards


(Chandukreddi) #3

Sorry Mario for delay.

beats: metricbeat-6.1.3-x86_64.rpm
OS: Red Hat Enterprise Linux Server release 7.4

Thanks
Chandra


(Chandukreddi) #4

Hello Experts,

Can anyone help me? I am new to metricsbeat and I stuck.

Thanks
Chandra


(Andrew Cholakian) #5

@chandukreddi can you share one of the Elasticsearch documents that are being created in metricbeat? It'd be useful to see exactly which fields are being populated.


(Chandukreddi) #6

Hi Andrew,

Sorry, some reason I am not getting alerts when topic get updated.

Here you go!

{
"_index": "cas-beat-2018.10.17-000002",
"_type": "doc",
"_id": "vS-gy2YBhjpsdWfhvk22",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-10-31T20:20:45.784Z",
"system": {
"process": {
"pgid": 74076,
"cmdline": "python /usr/local/dse-5.0.14/resources/cassandra/bin/cqlsh.py",
"cwd": "/root",
"fd": {
"open": 8,
"limit": {
"soft": 120000,
"hard": 120000
}
},
"state": "sleeping",
"username": "root",
"cpu": {
"total": {
"pct": 0.004,
"norm": {
"pct": 0.0001
}
},
"start_time": "2018-10-31T19:49:22.000Z"
},
"ppid": 70795,
"name": "python",
"memory": {
"size": 550658048,
"rss": {
"bytes": 35090432,
"pct": 0.0003
},
"share": 4681728
},
"pid": 74076
}
},
"metricset": {
"module": "system",
"name": "process",
"rtt": 83020
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
},
"fields": {
"@timestamp": [
"2018-10-31T20:20:45.784Z"
],
"system.process.cpu.start_time": [
"2018-10-31T19:49:22.000Z"
]
},
"sort": [
1541017245784
]
}


(Chandukreddi) #7

@Andrew_Cholakian1, Could you please help me?

Thanks
Chandra


(Andrew Cholakian) #8

So, the filesystem metricset in your config is configured to only send events once per minute. That means only some events will have filesystem data.

You can read more about that here.

In the document you shared I do see CPU stats, so that seems to be working correctly.


(Chandukreddi) #9

Thanks Andrew!

In case your referring to below lines, I believe that is related to that particular process right?

I am looking for overall host CPU usage and LOAD, on top that it also missing diskio, if you look at overall document it just captured only system module process info.

"system": {
"process": {

process:
"pgid": 74076,
"cmdline": "python /usr/local/dse-5.0.14/resources/cassandra/bin/cqlsh.py",

"cpu": {
"total": {
"pct": 0.004,
"norm": {
"pct": 0.0001
}
},
"start_time": "2018-10-31T19:49:22.000Z"
},


(Andrew Cholakian) #10

It looks like you're on windows, but windows doesn't offer a load statistic.


(Chandukreddi) #11

Hi Andrew,

its not windows, its Linux.

Thanks
Chandra


(Andrew Cholakian) #12

Do you have the system load metricset enabled? https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-load.html


(Chandukreddi) #13

Hi @Andrew_Cholakian1,

Yes I have enabled on sytem.yml file, Do I need to enable anywhere else? and I do see those fields in my Kibana when I disable option hide missing fields, this means its enabled but not capturing any data.

--- System module parameters

  • module: system

    period: 10s
    metricsets:

    • cpu
    • load
    • memory
    • network
    • process
    • process_summary
    • core
    • diskio

(Chandukreddi) #14

@all , Can anyone help me to understand why I can't see Load/CPU/DISK IO info in elasticsearch docs?

Thanks
Chandra


(Andrew Cholakian) #15

Hmmm, it should be showing up. I'm not sure why it isn't. @ruflin or @shaunak do you have any ideas here?


(Shaunak Kashyap) #16

Yes, it's odd that it's not showing up.

@chandukreddi Could you try the following (temporarily for debugging purposes only) to help and narrow down the issue?

  1. Disable all modules except system.
  2. Edit your ./modules.d/system.yml and only enable the cpu metricset.
  3. Start up Metricbeat but with the -e -d"*" flags. This will generate debugging output. Could you please post this here? Just the first 10 seconds or so of output should be sufficient.

Thanks,

Shaunak


(Chandukreddi) #17

Okay @shaunak let me try that.

Thanks
Chandra


(Chandukreddi) #18

Hi @shaunak
2018/11/19 22:47:16.180010 wrapper.go:149: DBG [module] Starting metricSetWrapper[module=system, name=diskio, host=]
2018/11/19 22:47:16.350588 processor.go:275: DBG [publish] Publish event: {
"@timestamp": "2018-11-19T22:47:16.180Z",
"@metadata": {
"beat": "metricbeat",
"type": "doc",
"version": "6.1.3"
},
"system": {
"diskio": {
"name": "sdg",
"read": {
"count": 137,
"time": 20,
"bytes": 2334720
},
"write": {
"bytes": 0,
"count": 0,
"time": 0
},
"io": {
"time": 17
},
"iostat": {
"service_time": 0,
"busy": 0,
"read": {
"request": {
"merges_per_sec": 0,
"per_sec": 0
},
"per_sec": {
"bytes": 0
}
},
"write": {
"request": {
"merges_per_sec": 0,
"per_sec": 0
},
"per_sec": {
"bytes": 0
}
},
"queue": {
"avg_size": 0
},
"request": {
"avg_size": 0
},
"await": 0
},
"serial_number": "36d094660757264002333866362b40e36"
}
},
"metricset": {
"module": "system",
"name": "diskio",
"rtt": 170174
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
}
Thanks
Chandra


(Chandukreddi) #19

My System module.
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/modules.d/system.yml

  • module: system

    period: 10s
    metricsets:

    • cpu

- load

- memory

- network

- process

- process_summary

- core

- diskio

#- socket

processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory

  • module: system
    period: 1m
    metricsets:

    • filesystem
    • fsstat
      processors:
    • drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  • module: system
    period: 15m
    metricsets:

    • uptime

Thanks
Chandra


(Chandukreddi) #20

Here is the CPU module stats

2018/11/19 22:47:15.825514 wrapper.go:149: DBG [module] Starting metricSetWrapper[module=system, name=cpu, host=]
2018/11/19 22:47:15.828325 processor.go:275: DBG [publish] Publish event: {
"@timestamp": "2018-11-19T22:47:15.825Z",
"@metadata": {
"beat": "metricbeat",
"type": "doc",
"version": "6.1.3"
},
"system": {
"cpu": {
"irq": {
"pct": 0
},
"softirq": {
"pct": 0
},
"user": {
"pct": 0
},
"idle": {
"pct": 0
},
"steal": {
"pct": 0
},
"system": {
"pct": 0
},
"iowait": {
"pct": 0
},
"total": {
"pct": 0
},
"nice": {
"pct": 0
},
"cores": 32
}
},
"metricset": {
"name": "cpu",
"rtt": 2472,
"module": "system"
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
}

Thanks
Chandra