# Metric older than 10 minutes **URL:** \ **Category:** Kibana\ **Created:** [February 22, 2020, 7:24am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472 "2020-02-22T07:24:22Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Reshad](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@Reshad](https://discuss.elastic.co/u/Reshad)\ **Post date:** [February 22, 2020, 7:24am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/1 "2020-02-22T07:24:22Z")
Hey, I am setting up a monitoring system for my infra and using metricbeat from my servers (3 for now). What I want is a visulization that shows me a list of hosts that have not checked in to ES in the past 10 minutes. This will ideally be a data-table with `host.hostname` and `@timestamp` where the timestamp is 10 min older than ether `now`. Creating a data-table of `host.hostname` to last `@timestamp` is pretty trivial, but I can't figure out how to filter the results so it only shows me records for timestamp older than 10 minutes. Any help will be really appreciated ---
**Author:** ![anjilinga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anjilinga/32/86010_2.png) [@anjilinga](https://discuss.elastic.co/u/anjilinga)\ **Post date:** [February 23, 2020, 7:01pm UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/2 "2020-02-23T19:01:58Z")
In bucket aggregation select as filters and give condion with the metric value to know the hosts that have not checked in to ES. and group by hosts using terms aggregation aggregation. in data table visulization. if possible pls provide more deatils of data. ---
**Author:** ![Reshad](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@Reshad](https://discuss.elastic.co/u/Reshad)\ **Post date:** [February 24, 2020, 5:09am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/3 "2020-02-24T05:09:11Z")
Hi Anji, Thanks for your help. The data I have is basic metricbeat data for 3 systems `metricbeat-1`, `metricbeat-2`, and `metricbeat-3`. I have pasted an example raw json for one log line here: ```auto {"_index":"metricbeat-7.5.2-2020.02.03-000001","_type":"_doc","_id":"PK9gdXABDxuLEUB7OTF3","_version":1,"_score":null,"_source":{"@timestamp":"2020-02-24T04:06:41.145Z","host":{"id":"33add492312d460eb3d8061e6502d713","containerized":false,"hostname":"metricbeat-1","architecture":"x86_64","name":"metricbeat-1","os":{"codename":"bionic","platform":"ubuntu","version":"18.04.4 LTS (Bionic Beaver)","family":"debian","name":"Ubuntu","kernel":"4.15.0-88-generic"}},"agent":{"type":"metricbeat","ephemeral_id":"c80e62c6-fb34-402e-8e99-4fbd634087ac","hostname":"metricbeat-1","id":"f1eed983-ed5b-43f8-b7c5-0c1ad4f7045e","version":"7.5.2"},"metricset":{"name":"memory","period":600000},"service":{"type":"system"},"system":{"memory":{"total":1033547776,"used":{"bytes":894615552,"pct":0.8656},"free":138932224,"actual":{"free":733540352,"used":{"bytes":300007424,"pct":0.2903}},"swap":{"used":{"bytes":1585152,"pct":0.0008},"free":2065154048,"in":{"pages":91},"out":{"pages":394},"readahead":{"cached":13,"pages":30},"total":2066739200},"hugepages":{"free":0,"reserved":0,"surplus":0,"default_size":2097152,"swap":{"out":{"pages":0,"fallback":0}},"total":0,"used":{"pct":0,"bytes":0}}}},"event":{"dataset":"system.memory","module":"system","duration":645037},"ecs":{"version":"1.1.0"}},"fields":{"@timestamp":["2020-02-24T04:06:41.145Z"]},"highlight":{"host.hostname":["@kibana-highlighted-field@metricbeat-1@/kibana-highlighted-field@"]},"sort":[1582517201145]} ``` This line is large, but the fields I care about are `host.hostname` and `timestamp` I have included some example data with just these fields below ```auto {"_source":{"@timestamp":"2020-02-24T04:05:00.000Z","host":{"hostname":"metricbeat-1"}}} {"_source":{"@timestamp":"2020-02-24T04:10:00.000Z","host":{"hostname":"metricbeat-1"}}} {"_source":{"@timestamp":"2020-02-24T04:15:00.000Z","host":{"hostname":"metricbeat-1"}}} {"_source":{"@timestamp":"2020-02-24T04:20:00.000Z","host":{"hostname":"metricbeat-1"}}} {"_source":{"@timestamp":"2020-02-24T04:25:00.000Z","host":{"hostname":"metricbeat-1"}}} {"_source":{"@timestamp":"2020-02-24T04:30:00.000Z","host":{"hostname":"metricbeat-1"}}} {"_source":{"@timestamp":"2020-02-24T04:05:00.000Z","host":{"hostname":"metricbeat-2"}}} {"_source":{"@timestamp":"2020-02-24T04:10:00.000Z","host":{"hostname":"metricbeat-2"}}} {"_source":{"@timestamp":"2020-02-24T04:15:00.000Z","host":{"hostname":"metricbeat-2"}}} {"_source":{"@timestamp":"2020-02-24T04:20:00.000Z","host":{"hostname":"metricbeat-2"}}} {"_source":{"@timestamp":"2020-02-24T04:25:00.000Z","host":{"hostname":"metricbeat-2"}}} {"_source":{"@timestamp":"2020-02-24T04:30:00.000Z","host":{"hostname":"metricbeat-2"}}} {"_source":{"@timestamp":"2020-02-24T04:05:00.000Z","host":{"hostname":"metricbeat-3"}}} {"_source":{"@timestamp":"2020-02-24T04:10:00.000Z","host":{"hostname":"metricbeat-3"}}} ``` These metrics are what kibana sees for the 3 systems. assuming the current time is `2020-02-24T04:31:00.000Z` I want to make a dashboard that show me host `metricbeat-3` is down (has not conected to the ES server in the last 10 minutes). I can't simply filter for timestamp `now-10m/m` because all 3 hosts have a log that matches that time filter. If I do this I end up with a table that looks like: ```auto hostame | timestamp metricbeat-1 | 2020-02-24T04:20:00.000Z metricbeat-2 | 2020-02-24T04:20:00.000Z metricbeat-3 | 2020-02-24T04:10:00.000Z ``` What I need to do is exclude buckets where any timestamp (the latest one) is less than `now-10m/m` from the table. ---
**Author:** ![anjilinga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anjilinga/32/86010_2.png) [@anjilinga](https://discuss.elastic.co/u/anjilinga)\ **Post date:** [February 25, 2020, 9:26am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/4 "2020-02-25T09:26:22Z")
Hi Reshad, if you have these two fields only in your data, you can create one scripted field by calculating the time difference. based on that scripted field you can filter the data. I am checking the options mean while you can try with scripted field. ---
**Author:** ![Reshad](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@Reshad](https://discuss.elastic.co/u/Reshad)\ **Post date:** [February 25, 2020, 1:45pm UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/5 "2020-02-25T13:45:19Z")
Hi Anji, Thanks for your help. I did some reading and what I want seems to be a scripted field (called `time_difference`) that takes `timestamp` and calculates the difference (in minutes or even seconds) to `now`. The table then becomes a simple all hostnames with `time_difference` greater than 10 (or 600 if seconds) However `now` does not seem to be supported in the `painless` scripting language. [https://www.elastic.co/guide/en/elasticsearch/painless/7.6/painless-datetime.html#\_datetime\_now](https://www.elastic.co/guide/en/elasticsearch/painless/7.6/painless-datetime.html#_datetime_now) Is there something I am missing? I am attempting this using the Kibana UI (under Management \> Kibana \> Index Patterns \> metricbeat-\* \> scripted fields \> add scripted field Should I be using some other way to do this? ---
**Author:** ![anjilinga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anjilinga/32/86010_2.png) [@anjilinga](https://discuss.elastic.co/u/anjilinga)\ **Post date:** [February 25, 2020, 8:58pm UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/6 "2020-02-25T20:58:18Z")
Hi Could you please try using the below. please use the below to get the current date time. int docminutes = doc['@timestamp'].value.minute; Calendar rightNow = Calendar.getInstance(); int currentminutes = rightNow.get(Calendar.MINUTE); if(docminutes-currentminutes \>10) { return true; } else{ return false; } you please change the logic as per your requirements. ---
**Author:** ![Reshad](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@Reshad](https://discuss.elastic.co/u/Reshad)\ **Post date:** [February 28, 2020, 9:47am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/7 "2020-02-28T09:47:10Z")
Hi, Thanks for this help I have created 3 new fields `current_timestamp`, `timestamp_epoch`, and `time_diff` and have attached them below. current\_timestamp: ```auto /* * This script creates a new filed with the current system time. * The output of this script is added to each document on the fly when it is searched. * * See: * - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Calendar.html * - https://docs.oracle.com/javase/8/docs/api/java/text/DateFormat.html */ // Create a new calendar object with the current time Calendar time_now = Calendar.getInstance(); // Setup formatting (for ISO 8601 date format) DateFormat iso = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSX"); // Return the date in ISO 8601 format return iso.format(time_now.getTime()); ``` timestamp\_epoch: ```auto /* * This script creates a new filed with the timestamp in the doc as Unix Epoch. * The output of this script is added to each document on the fly when it is searched. * * See: * - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Calendar.html */ // Create a new calendar object with the current time for now and the time of the document Calendar time_doc = Calendar.getInstance(); // Use the document to set the time in the object time_doc.set(Calendar.YEAR, doc['@timestamp'].value.year); time_doc.set(Calendar.MONTH, (doc['@timestamp'].value.monthOfYear - 1)); // Java used 0 for January time_doc.set(Calendar.DAY_OF_MONTH, doc['@timestamp'].value.dayOfMonth); time_doc.set(Calendar.HOUR, doc['@timestamp'].value.hour); time_doc.set(Calendar.MINUTE, doc['@timestamp'].value.minute); time_doc.set(Calendar.SECOND, doc['@timestamp'].value.second); time_doc.set(Calendar.MILLISECOND, doc['@timestamp'].value.millisOfSecond); // Return the time difference in seconds return (time_doc.getTimeInMillis()) / 1000; ``` time\_diff: ```auto /* * This script creates a new filed with the difference between current system time and the timestamp in the doc. * The output of this script is added to each document on the fly when it is searched. * * See: * - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Calendar.html */ // Create a new calendar object with the current time for now and the time of the document Calendar time_now = Calendar.getInstance(); Calendar time_doc = Calendar.getInstance(); // Use the document to set the time in the object time_doc.set(Calendar.YEAR, doc['@timestamp'].value.year); time_doc.set(Calendar.MONTH, (doc['@timestamp'].value.monthOfYear - 1)); // Java used 0 for January time_doc.set(Calendar.DAY_OF_MONTH, doc['@timestamp'].value.dayOfMonth); time_doc.set(Calendar.HOUR, doc['@timestamp'].value.hour); time_doc.set(Calendar.MINUTE, doc['@timestamp'].value.minute); time_doc.set(Calendar.SECOND, doc['@timestamp'].value.second); time_doc.set(Calendar.MILLISECOND, doc['@timestamp'].value.millisOfSecond); // Return the time difference in seconds return (time_now.getTimeInMillis() - time_doc.getTimeInMillis()) / 1000; ``` I still am not able to figure out how to get a data-table of `host.hostname`s where the document with the max `@timestamp` has a `time_diff` greater than `600` (10 minutes) ---
**Author:** ![Reshad](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@Reshad](https://discuss.elastic.co/u/Reshad)\ **Post date:** [February 28, 2020, 11:31am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/8 "2020-02-28T11:31:15Z")
Hey I solved this by doing the following: 1. create a transform which makes an index that contains ONLY documents with the latest timestamp per hostname ```auto PUT _transform/last-heartbeat-per-hostname { "source": { "index": [ "metricbeat-*" ] }, "pivot": { "group_by": { "host.hostname": { "terms": { "field": "host.hostname" } } }, "aggregations": { "@timestamp": { "max": { "field": "@timestamp" } } } }, "description": "last-heartbeat-per-hostname", "dest": { "index": "last-heartbeat-per-hostname" }, "sync": { "time": { "field": "@timestamp", "delay": "60s" } } } ``` 1. Start the transform ```auto POST _transform/last-heartbeat-per-hostname/_start ``` 1. Create a Kibana index called `last-heartbeat-per-hostname` (which matches the transform destination index). Use `@timestamp` as the time filter field name. 2. Create a new `Data Table` Visualization with the following parameters: 3. Create a dashboard ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/d/0d3f204173917a4d194a797bbda905ad61b9d6c9.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 27, 2020, 11:31am UTC](https://discuss.elastic.co/t/metric-older-than-10-minutes/220472/9 "2020-03-27T11:31:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.