Hello Team,
I have enabled system module in metricbeat but all I see is Processora details in Elasticsearch but I don;t find anything CPU/CORE/DISK related information captured. any idea?
system.yml
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/modules.d/system.yml
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
- core
- diskio
#- socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
- module: system
period: 15m
metricsets:
- uptime
**metricbeat.yml**
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/metricbeat.yml | grep setup
setup.template.name: "cassandra-beats"
setup.template.pattern: "cas-beats-*"
setup.template.settings:
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
#setup.dashboards.url:
setup.kibana:
# `setup.kibana.host` options.
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/metricbeat.yml | grep elastic -A10
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.1.27.6:9200"]
index: "logs_write_cas_beat"
#template: "/etc/metricbeat/cass_beat_2.json"
#template_name: "cassandra_beat"
#template_overwrite: true
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
Thanks
Chandra
Hi @chandukreddi
Can you specify your metricbeat version and you OS? I guess is some kind of Linux
Best regards
Sorry Mario for delay.
beats: metricbeat-6.1.3-x86_64.rpm
OS: Red Hat Enterprise Linux Server release 7.4
Thanks
Chandra
Hello Experts,
Can anyone help me? I am new to metricsbeat and I stuck.
Thanks
Chandra
@chandukreddi can you share one of the Elasticsearch documents that are being created in metricbeat? It'd be useful to see exactly which fields are being populated.
Hi Andrew,
Sorry, some reason I am not getting alerts when topic get updated.
Here you go!
{
"_index": "cas-beat-2018.10.17-000002",
"_type": "doc",
"_id": "vS-gy2YBhjpsdWfhvk22",
"_version": 1,
"_score": null,
"_source": {
"@timestamp ": "2018-10-31T20:20:45.784Z",
"system": {
"process": {
"pgid": 74076,
"cmdline": "python /usr/local/dse-5.0.14/resources/cassandra/bin/cqlsh.py",
"cwd": "/root",
"fd": {
"open": 8,
"limit": {
"soft": 120000,
"hard": 120000
}
},
"state": "sleeping",
"username": "root",
"cpu": {
"total": {
"pct": 0.004,
"norm": {
"pct": 0.0001
}
},
"start_time": "2018-10-31T19:49:22.000Z"
},
"ppid": 70795,
"name": "python",
"memory": {
"size": 550658048,
"rss": {
"bytes": 35090432,
"pct": 0.0003
},
"share": 4681728
},
"pid": 74076
}
},
"metricset": {
"module": "system",
"name": "process",
"rtt": 83020
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
},
"fields": {
"@timestamp ": [
"2018-10-31T20:20:45.784Z"
],
"system.process.cpu.start_time": [
"2018-10-31T19:49:22.000Z"
]
},
"sort": [
1541017245784
]
}
@Andrew_Cholakian1 , Could you please help me?
Thanks
Chandra
So, the filesystem metricset in your config is configured to only send events once per minute. That means only some events will have filesystem data.
You can read more about that here.
In the document you shared I do see CPU stats, so that seems to be working correctly.
Thanks Andrew!
In case your referring to below lines, I believe that is related to that particular process right?
I am looking for overall host CPU usage and LOAD, on top that it also missing diskio, if you look at overall document it just captured only system module process info.
"system": {
"process": {
process:
"pgid": 74076,
"cmdline": "python /usr/local/dse-5.0.14/resources/cassandra/bin/cqlsh.py",
"cpu": {
"total": {
"pct": 0.004,
"norm": {
"pct": 0.0001
}
},
"start_time": "2018-10-31T19:49:22.000Z"
},
It looks like you're on windows, but windows doesn't offer a load statistic.
Hi Andrew,
its not windows, its Linux.
Thanks
Chandra
Hi @Andrew_Cholakian1 ,
Yes I have enabled on sytem.yml file, Do I need to enable anywhere else? and I do see those fields in my Kibana when I disable option hide missing fields, this means its enabled but not capturing any data.
--- System module parameters
module: system
period: 10s
metricsets:
cpu
load
memory
network
process
process_summary
core
diskio
@all , Can anyone help me to understand why I can't see Load/CPU/DISK IO info in elasticsearch docs?
Thanks
Chandra
Hmmm, it should be showing up. I'm not sure why it isn't. @ruflin or @shaunak do you have any ideas here?
shaunak
(Shaunak Kashyap)
November 19, 2018, 4:57pm
16
Yes, it's odd that it's not showing up.
@chandukreddi Could you try the following (temporarily for debugging purposes only) to help and narrow down the issue?
Disable all modules except system
.
Edit your ./modules.d/system.yml
and only enable the cpu
metricset.
Start up Metricbeat but with the -e -d"*"
flags. This will generate debugging output. Could you please post this here? Just the first 10 seconds or so of output should be sufficient.
Thanks,
Shaunak
Okay @shaunak let me try that.
Thanks
Chandra
Hi @shaunak
2018/11/19 22:47:16.180010 wrapper.go:149: DBG [module] Starting metricSetWrapper[module=system, name=diskio, host=]
2018/11/19 22:47:16.350588 processor.go:275: DBG [publish] Publish event: {
"@timestamp ": "2018-11-19T22:47:16.180Z",
"@metadata ": {
"beat": "metricbeat",
"type": "doc",
"version": "6.1.3"
},
"system": {
"diskio": {
"name": "sdg",
"read": {
"count": 137,
"time": 20,
"bytes": 2334720
},
"write": {
"bytes": 0,
"count": 0,
"time": 0
},
"io": {
"time": 17
},
"iostat": {
"service_time": 0,
"busy": 0,
"read": {
"request": {
"merges_per_sec": 0,
"per_sec": 0
},
"per_sec": {
"bytes": 0
}
},
"write": {
"request": {
"merges_per_sec": 0,
"per_sec": 0
},
"per_sec": {
"bytes": 0
}
},
"queue": {
"avg_size": 0
},
"request": {
"avg_size": 0
},
"await": 0
},
"serial_number": "36d094660757264002333866362b40e36"
}
},
"metricset": {
"module": "system",
"name": "diskio",
"rtt": 170174
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
}
Thanks
Chandra
My System module.
Dell-TMO-CT-Cass1:/root>cat /etc/metricbeat/modules.d/system.yml
module: system
period: 10s
metricsets:
- load
- memory
- network
- process
- process_summary
- core
- diskio
#- socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
Thanks
Chandra
Here is the CPU module stats
2018/11/19 22:47:15.825514 wrapper.go:149: DBG [module] Starting metricSetWrapper[module=system, name=cpu, host=]
2018/11/19 22:47:15.828325 processor.go:275: DBG [publish] Publish event: {
"@timestamp ": "2018-11-19T22:47:15.825Z",
"@metadata ": {
"beat": "metricbeat",
"type": "doc",
"version": "6.1.3"
},
"system": {
"cpu": {
"irq": {
"pct": 0
},
"softirq": {
"pct": 0
},
"user": {
"pct": 0
},
"idle": {
"pct": 0
},
"steal": {
"pct": 0
},
"system": {
"pct": 0
},
"iowait": {
"pct": 0
},
"total": {
"pct": 0
},
"nice": {
"pct": 0
},
"cores": 32
}
},
"metricset": {
"name": "cpu",
"rtt": 2472,
"module": "system"
},
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.1.3"
}
}
Thanks
Chandra