This article is also available in Italian.
Lens is a relatively new tool in Kibana, which went GA less than a year ago as universal visualization editor, and already gained some interest for its advanced features minor after minor. Being one of the many tools in Kibana it is easy to miss or forget about some of new nice additions, so in this post we'll revisit some of the advanced use cases of the tool.
I've decided to write down here some of the most interesting use cases recently I've seen solved by these new features.
Taming missing values
Not all records come filled completely, sometimes by design with optional properties, other times for other reasons out of our control (a reset/timeout of a monitoring system, app restart, etc...). Having them as aggregation results, then in your visualizations can be annoying.
Here's a collection of various solutions offered by Lens to tame such problem.
What options to represent missing values on charts?
When using Line or Area charts in Lens missing values are hidden by default.
When hiding is not the desired solution, it is possible to choose between a set of other interpolation options:
Difference from previous bucket: what if it's an empty value?
One possible solution to compute a bucket difference is the differences function:
differences( max(fieldA ))
or a more sophisticated Formula using last_value:
differences(last_values( fieldA ))
While these formulas work, missing values are skipped by default, which may lead to misleading results with negative buckets.
Lens provides a special function to handle use cases where positive only rates are required, named Counter rate:
The rate can be tuned on the panel to the desired unit.
Build a table showing only last "existing" value
Table visualization is one of the most basic, yet useful, way to represent data in Lens: it can be used to visualize multiple aggregations as row, columns or metrics.
In this scenario there's the need to display, as metric, on the table the last value for a particular field:
The table above shows all last values for each time bucket (1 bucket - 12 hours ): for each bucket records are sorted by a date field, and the value of the last one is sent (as you might expect from a function "Last value").
In the specific use case only records with a valid value were targeted, therefore a filter had to be applied before "Last value" got applied: luckily Lens supports filters on the specific metric which could be take advantage in this case.
Opening the Last value metric configuration, from the Add advanced options click the Filter by to expose the feature, and apply the required filter:
It is possible to use either KQL or Lucene syntax to define the desired filter.
Future work
As Lens team we're aware that there's still some more work to be done for missing value handling, and some improvement use cases have been already found, starting from visual optimizations to specific per-metric handlers for missing values.
Formula comparisons
Compare two values, either static or dynamic, in order to return a result is a frequent scenario in the analytics world. In this section we'll discover how to use math operators to achieve comparison-like result.
Comparing against a static value
Let's consider the following example to explain this case:
we're looking for a way to report a metric, only when the this gets above the value of 1000
We know values can only be integers, so some assumptions can be made here to model the problem:
- if
metric >= 1000returnmetric - else return 0
The trick in Formula is to use the clamp function available in Tinymath to limit the metric value between 0 and 1:
clamp( metric(field) - 1000, 0, 1) * metric(field)
In the formula above the conditional part has been codified as:
- if
metric - 1000 is positivethen return1 - otherwise return
0
The conditional result is then multiplied by the metric itself for the requested result.
Because of the strong assumption about integer values we can assume that no value will fall within 0 and 1 used in clamp. Mind that as long as we know that no value falls within the (0-1) range, also fractional values can be used safely.
Comparing two dynamic values
In this scenario both sides of the comparison are results of a dynamic aggregation of the data:
we're looking to report the biggest metric value between A and B
Also in this case the trick involves clamp but the formula is a bit different:
clamp( metric(fieldA) - metric(fieldB), 0, 1) * metric(fieldA) +
clamp( metric(fieldB) - metric(fieldA), 0, 1) * metric(fieldB)
The clamping is repeated twice in this case, inverting the order of the difference operators, summing up the final result: because the difference can be positive only on one side the possible results are either 1 * metric(fieldA) + 0 or 0 + 1 * metric(fieldB).
Future work
There's already the idea to extend the current Lens Formula syntax with conditional/comparison operators which will make it easier to have lower level control over values and better error handling for edge cases.