Metricbeat can't make visualization due to null value in index fields value

[PRASHANT_MEHTA]

Hello All,

I am unable to create TSVB visulization TOP N in kibana,as I'm trying to display system.filesystem.used.pct of mount point but both these fiels have null values(-) in index.
Any suggestion would be helpful!

Thanx,
Prashant

[stephenb (Stephen Brown)]

Hi @PRASHANT_MEHTA

Couple items... I am a little confused.

What version are you on?

What are you actually try to show?

That is a Table View not Top N

I would put the following in the panel filter so only file system documents show up.

event.dataset : "system.filesystem"

[PRASHANT_MEHTA]

Hello @stephenb ,

Thanks for looking into this!
I'm using kibana,metricbeat,elastic 7.9.1 version.I'm trying to implement a use case through
TSVM(option TOP N).Which would display only those mount points which have reached 90%
utilization and dont require below 90%.
So I'm implementing this using TOP N ,but challenge I face is that I need to display hostName also.Currently only filesystem used % and filesystem name can be seen,but I need the hostname
as well.This cant be achived in kibana directly I guess,so I'm trying to append the hostname with
mountpoint name through javascript processor,but host name as an event is not picked ,I'm not sure why.
Even sometimes visulization shows wrong data like 0% and green bar.Like below,Ideally through panel filter I will filter system.filesystem.used.pct > 0.9.....But this cant be seen at time and at time it does....Seems bug.

SEEMS BUG

Aggregation:

Correct Requirement:[mountpoint hostname---------------------filesystem used % value]

Below Image- only red bar should be shown as alert which can be achived through panel filter > 0.9(90%),as I cant show three values in visulization so I am trying to create new field that would concatenate(mountpoint- hostname---------------------------------file.system.used.pct)

Can u assist how can this be best achived and why JS processor in metricbeat dosent't pick up hostname event?

var console = require('console');
function process(event) {


     var fileSystemUsedPercent = event.Get("system.filesystem.used.pct");//filesystem used percentage
     console.info("File system used percentage value [%s]", fileSystemUsedPercent);
     var mountPoint = event.Get("system.filesystem.mount_point");  //mountpoint name                                    
     console.info("mountPoint [%s]", mountPoint);
     var hostName = event.Get(host.name);       //hostname [DOSENT PICK UP HOST EVENT THOUGH PRESENT IN INDEX]                              
     console.info("hostName [%s]", hostName);
     if(fileSystemUsedPercent > 0.8) {
		     event.Put("filesystem.status","critical");
         event.Put("filesystem.mount.host",mountPoint + ' ' + hostName);//display hostname and mountpoint if greater than treshold
         console.info("filesystem.status [%s]", event.Get("filesystem.status"));
     } 
     
     
     return event;
 }

Many Thanks,
Prashant

[stephenb (Stephen Brown)]

Hi @PRASHANT_MEHTA

Lets back up a bit.

1st 7.9 is ancient / EOL and I had to set up a 7.9.3 stack because so much has changed.... if you were on a recent release there would be some pretty simple ways to solve this.

I think what you want to do is

"As an ops user I was display the Filesystems with the Highest system.filesystem.used.pct and show them sorted by max and display both host.name and system.filesystem.mount_point"

Ok let/s work on that.

1st The Top Hits aggregation you are trying to use is absolutely the wrong aggregation... Top Hits means "Most Common" not Max Value so that will never work your aggregation is asking for the average of the most common value.. and that is why the graph looks wrong .

So First lets see if we can get the Top N working first (without the hostname) just for learning purposes..

Panel Options, Says use the last value (but drop the last partial bucket)
Make sure the bucket size / interval >=1min (the collection rate) and filter on the data set.

Now find the Average of the Filesystem % and sort them Descending

And set to Percent

That is the correct way to build that...

Try that and make sure it works, come back when you have it working ... then perhaps we can figure out the concatenate the host + mount point. I am not familiar with the Java Script processor. (again in a new stack this is a 5 min fix)

[stephenb (Stephen Brown)]

in 7.9.3. You can get what you want by using a data table
You need to hit "update" after you make each change

Select metricbeat-*

Select your Metric

Split Rows By Host

The mount point

And you should get this

You can use custom labels to make it look nicer and sorry my data is boring.

[PRASHANT_MEHTA]

@stephenb Thanx for helping in this! It worked fine.Can you please suggest ELK version above
7.9.1 i.e not log4j vulnerable and non-paid/open source version that can be used?

Many Thanks,
Prashant

[stephenb (Stephen Brown)]

I would use 7.17.3 or 8.2 with the Elastic Basic License which is Free
You can see the features here under the FREE AND OPEN - BASIC Heading
Once you are on a newer version I would highly recommend looking at / using Lens

@PRASHANT_MEHTA Ohh the new version would solve your other issues of concatenating the host + filesystem... 5 minute task with new runtime fields...

[PRASHANT_MEHTA]

Hello All,

Probable solution found and here is the code and system.yml for reference.

/**
The ProcessDataUpdater implements to combine the host name, process name and process id data in a separate filed.
*/

var console = require('console');

var params = {hostname: ""};

/**
Gets the script parameter passed from the system.yml file. With this context, host name is passed as input.
*/
function register(scriptParams) {
	params = scriptParams;
}
	
/**
Entry point to run the logic. It gets the host name, process name and process id form the current event and creates a new field with combining the those values.

Also, it returns the updated event.
*/	
function process(event) {
	
	var hostName = params.hostname;
	
	var processId = event.Get("process.pid");
	
	var processName = event.Get("process.name");
	
	event.Put("process.details", hostName + "_" + processName + "_" + processId)
	
	var processDetails = event.Get("process.details");
	
	return event;
}

system.yml

# Module: system
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.8/metricbeat-module-system.html
- module: system
  period: 10s
  metricsets:
    - load
    - memory
    - network

- module: system
  period: 60s
  metricsets:
    - fsstat
    - filesystem
  filesystem.ignore_types: [ssysfs, rootfs, ramfs, bdev, cgroup, cpuset,debugfs, securityfs, sockfs, dax, bpf, pipefs, anon_inodefs, configfs, devpts, hugetlbfs, autofs, pstore, mqueue, selinuxfs, rpc_pipefs, binfmt_misc, overlay,nfs4,nfs,proc,sysfs]

        
- module: system
  period: 10s
  metricsets:
    - diskio
  diskio.include_devices: ["sda", "sda1","vda","vda1"]
    
- module: system
  metricsets:
    - process
  processes: ['.*']
  process.include_cpu_ticks: true
  processors:
  - script:
        lang: javascript
        id: ProcessDataUpdater
        file: ProcessDataUpdater.js
        params:
            hostname: '${HOSTNAME}'

- module: system
  period: 10s
  metricsets:
    - cpu
  processors:
  - script:
        lang: javascript
        id: CPUStatusUpdater.js
        file: CPUStatusUpdater.js