Metric older than 10 minutes

Reshad · February 22, 2020, 7:24am

Hey,

I am setting up a monitoring system for my infra and using metricbeat from my servers (3 for now).
What I want is a visulization that shows me a list of hosts that have not checked in to ES in the past 10 minutes.
This will ideally be a data-table with host.hostname and @timestamp where the timestamp is 10 min older than ether now.

Creating a data-table of host.hostname to last @timestamp is pretty trivial, but I can't figure out how to filter the results so it only shows me records for timestamp older than 10 minutes.

Any help will be really appreciated

anjilinga · February 23, 2020, 7:01pm

In bucket aggregation select as filters and give condion with the metric value to know the hosts that have not checked in to ES. and group by hosts using terms aggregation aggregation. in data table visulization.

if possible pls provide more deatils of data.

Reshad · February 24, 2020, 5:09am

Hi Anji,

Thanks for your help.

The data I have is basic metricbeat data for 3 systems metricbeat-1, metricbeat-2, and metricbeat-3.

I have pasted an example raw json for one log line here:

{"_index":"metricbeat-7.5.2-2020.02.03-000001","_type":"_doc","_id":"PK9gdXABDxuLEUB7OTF3","_version":1,"_score":null,"_source":{"@timestamp":"2020-02-24T04:06:41.145Z","host":{"id":"33add492312d460eb3d8061e6502d713","containerized":false,"hostname":"metricbeat-1","architecture":"x86_64","name":"metricbeat-1","os":{"codename":"bionic","platform":"ubuntu","version":"18.04.4 LTS (Bionic Beaver)","family":"debian","name":"Ubuntu","kernel":"4.15.0-88-generic"}},"agent":{"type":"metricbeat","ephemeral_id":"c80e62c6-fb34-402e-8e99-4fbd634087ac","hostname":"metricbeat-1","id":"f1eed983-ed5b-43f8-b7c5-0c1ad4f7045e","version":"7.5.2"},"metricset":{"name":"memory","period":600000},"service":{"type":"system"},"system":{"memory":{"total":1033547776,"used":{"bytes":894615552,"pct":0.8656},"free":138932224,"actual":{"free":733540352,"used":{"bytes":300007424,"pct":0.2903}},"swap":{"used":{"bytes":1585152,"pct":0.0008},"free":2065154048,"in":{"pages":91},"out":{"pages":394},"readahead":{"cached":13,"pages":30},"total":2066739200},"hugepages":{"free":0,"reserved":0,"surplus":0,"default_size":2097152,"swap":{"out":{"pages":0,"fallback":0}},"total":0,"used":{"pct":0,"bytes":0}}}},"event":{"dataset":"system.memory","module":"system","duration":645037},"ecs":{"version":"1.1.0"}},"fields":{"@timestamp":["2020-02-24T04:06:41.145Z"]},"highlight":{"host.hostname":["@kibana-highlighted-field@metricbeat-1@/kibana-highlighted-field@"]},"sort":[1582517201145]}

This line is large, but the fields I care about are host.hostname and timestamp I have included some example data with just these fields below

{"_source":{"@timestamp":"2020-02-24T04:05:00.000Z","host":{"hostname":"metricbeat-1"}}}
{"_source":{"@timestamp":"2020-02-24T04:10:00.000Z","host":{"hostname":"metricbeat-1"}}}
{"_source":{"@timestamp":"2020-02-24T04:15:00.000Z","host":{"hostname":"metricbeat-1"}}}
{"_source":{"@timestamp":"2020-02-24T04:20:00.000Z","host":{"hostname":"metricbeat-1"}}}
{"_source":{"@timestamp":"2020-02-24T04:25:00.000Z","host":{"hostname":"metricbeat-1"}}}
{"_source":{"@timestamp":"2020-02-24T04:30:00.000Z","host":{"hostname":"metricbeat-1"}}}
{"_source":{"@timestamp":"2020-02-24T04:05:00.000Z","host":{"hostname":"metricbeat-2"}}}
{"_source":{"@timestamp":"2020-02-24T04:10:00.000Z","host":{"hostname":"metricbeat-2"}}}
{"_source":{"@timestamp":"2020-02-24T04:15:00.000Z","host":{"hostname":"metricbeat-2"}}}
{"_source":{"@timestamp":"2020-02-24T04:20:00.000Z","host":{"hostname":"metricbeat-2"}}}
{"_source":{"@timestamp":"2020-02-24T04:25:00.000Z","host":{"hostname":"metricbeat-2"}}}
{"_source":{"@timestamp":"2020-02-24T04:30:00.000Z","host":{"hostname":"metricbeat-2"}}}
{"_source":{"@timestamp":"2020-02-24T04:05:00.000Z","host":{"hostname":"metricbeat-3"}}}
{"_source":{"@timestamp":"2020-02-24T04:10:00.000Z","host":{"hostname":"metricbeat-3"}}}

These metrics are what kibana sees for the 3 systems. assuming the current time is 2020-02-24T04:31:00.000Z I want to make a dashboard that show me host metricbeat-3 is down (has not conected to the ES server in the last 10 minutes).

I can't simply filter for timestamp now-10m/m because all 3 hosts have a log that matches that time filter.
If I do this I end up with a table that looks like:

hostame         |   timestamp
metricbeat-1    |   2020-02-24T04:20:00.000Z
metricbeat-2    |   2020-02-24T04:20:00.000Z
metricbeat-3    |   2020-02-24T04:10:00.000Z

What I need to do is exclude buckets where any timestamp (the latest one) is less than now-10m/m from the table.

anjilinga · February 25, 2020, 9:26am

Hi Reshad,

if you have these two fields only in your data, you can create one scripted field by calculating the time difference. based on that scripted field you can filter the data.

I am checking the options mean while you can try with scripted field.

Reshad · February 25, 2020, 1:45pm

Hi Anji,

Thanks for your help.

I did some reading and what I want seems to be a scripted field (called time_difference) that takes timestamp and calculates the difference (in minutes or even seconds) to now.
The table then becomes a simple all hostnames with time_difference greater than 10 (or 600 if seconds)

However now does not seem to be supported in the painless scripting language.
https://www.elastic.co/guide/en/elasticsearch/painless/7.6/painless-datetime.html#_datetime_now

Is there something I am missing?

I am attempting this using the Kibana UI (under Management > Kibana > Index Patterns > metricbeat-* > scripted fields > add scripted field
Should I be using some other way to do this?

anjilinga · February 25, 2020, 8:58pm

Hi

Could you please try using the below.

please use the below to get the current date time.
int docminutes = doc['@timestamp'].value.minute;
Calendar rightNow = Calendar.getInstance();
int currentminutes = rightNow.get(Calendar.MINUTE);
if(docminutes-currentminutes >10)
{
return true;
}
else{
return false;
}

you please change the logic as per your requirements.

Reshad · February 28, 2020, 9:47am

Hi,

Thanks for this help I have created 3 new fields current_timestamp, timestamp_epoch, and time_diff and have attached them below.

current_timestamp:

/*
*  This script creates a new filed with the current system time.
*  The output of this script is added to each document on the fly when it is searched.
*
*  See:
*  - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Calendar.html
*  - https://docs.oracle.com/javase/8/docs/api/java/text/DateFormat.html
*/

// Create a new calendar object with the current time
Calendar time_now = Calendar.getInstance();

// Setup formatting (for ISO 8601 date format)
DateFormat iso = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSX");

// Return the date in ISO 8601 format
return iso.format(time_now.getTime());

timestamp_epoch:

/*
*  This script creates a new filed with the timestamp in the doc as Unix Epoch.
*  The output of this script is added to each document on the fly when it is searched.
*
*  See:
*  - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Calendar.html
*/

// Create a new calendar object with the current time for now and the time of the document
Calendar time_doc = Calendar.getInstance();

// Use the document to set the time in the object
time_doc.set(Calendar.YEAR, doc['@timestamp'].value.year);
time_doc.set(Calendar.MONTH, (doc['@timestamp'].value.monthOfYear - 1)); // Java used 0 for January
time_doc.set(Calendar.DAY_OF_MONTH, doc['@timestamp'].value.dayOfMonth);
time_doc.set(Calendar.HOUR, doc['@timestamp'].value.hour);
time_doc.set(Calendar.MINUTE, doc['@timestamp'].value.minute);
time_doc.set(Calendar.SECOND, doc['@timestamp'].value.second);
time_doc.set(Calendar.MILLISECOND, doc['@timestamp'].value.millisOfSecond);

// Return the time difference in seconds
return (time_doc.getTimeInMillis()) / 1000;

time_diff:

/*
*  This script creates a new filed with the difference between current system time and the timestamp in the doc.
*  The output of this script is added to each document on the fly when it is searched.
*
*  See:
*  - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Calendar.html
*/

// Create a new calendar object with the current time for now and the time of the document
Calendar time_now = Calendar.getInstance();
Calendar time_doc = Calendar.getInstance();

// Use the document to set the time in the object
time_doc.set(Calendar.YEAR, doc['@timestamp'].value.year);
time_doc.set(Calendar.MONTH, (doc['@timestamp'].value.monthOfYear - 1)); // Java used 0 for January
time_doc.set(Calendar.DAY_OF_MONTH, doc['@timestamp'].value.dayOfMonth);
time_doc.set(Calendar.HOUR, doc['@timestamp'].value.hour);
time_doc.set(Calendar.MINUTE, doc['@timestamp'].value.minute);
time_doc.set(Calendar.SECOND, doc['@timestamp'].value.second);
time_doc.set(Calendar.MILLISECOND, doc['@timestamp'].value.millisOfSecond);

// Return the time difference in seconds
return (time_now.getTimeInMillis() - time_doc.getTimeInMillis()) / 1000;

I still am not able to figure out how to get a data-table of host.hostnames where the document with the max @timestamp has a time_diff greater than 600 (10 minutes)

Reshad · February 28, 2020, 11:31am

Hey I solved this by doing the following:

create a transform which makes an index that contains ONLY documents with the latest timestamp per hostname

PUT _transform/last-heartbeat-per-hostname
{
  "source": {
    "index": [
      "metricbeat-*"
    ]
  },
  "pivot": {
    "group_by": {
      "host.hostname": {
        "terms": {
          "field": "host.hostname"
        }
      }
    },
    "aggregations": {
      "@timestamp": {
        "max": {
          "field": "@timestamp"
        }
      }
    }
  },
  "description": "last-heartbeat-per-hostname",
  "dest": {
    "index": "last-heartbeat-per-hostname"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  }
}

Start the transform

POST _transform/last-heartbeat-per-hostname/_start

Create a Kibana index called last-heartbeat-per-hostname (which matches the transform destination index). Use @timestamp as the time filter field name.
Create a new Data Table Visualization with the following parameters:
- Use the index last-heartbeat-per-hostname as the source.
- Add a metric aggregation type Max for the field @timestamp, label it Last heartbeat
- Add a bucket to Split rows which uses
  - The aggregation Terms
  - Aggregates on the field host.hostname
  - Ordered by the Metric: Last heartbeat
  - Order Ascending (oldest heartbeat on top)
  - Size 15 (show top 15 systems that are down)
  - Enable grouping other values in a field called More...
  - Set a Label for the filed Hostname
- Add a filter with the following details:
  - Filed is @timestamp
  - Operator is is not between
  - from now-10m/m
  - to now
- Save the visualization as Hosts down for more than 10 minutes
Create a dashboard
- Add the visualization Hosts down for more than 10 minutes
- Click on the settings icon and customize time range
- Select an absolute date before you started logging and now as your start and end date
- Save the dashboard

system · March 27, 2020, 11:31am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.