Also available in Spanish
tl;dr;
This post shows how you can use the Elastic Stack to map IoT, Security or Observability data. The data that is collected in these use-cases often do not explicitly contain a latitude or longitude. It is still possible to map this data in Kibana by using a “Term Join”.
Mapping measures with locations
The Elastic Stack is uniquely suited for many Observability and Security use-cases, particularly if there is an IoT-component:
- Mapping private IP-addresses to an actual datacenter or office building
- Security events like entry-card swipes for employees entering or leaving a building or turnstile-flips
- Temperature or humidity sensors in different sections of a plant
- Robot-arm health metrics on a shopfloor
In all these cases, the actual object is stationary and uniquely identifiable (this particular temperature sensor, that particular turnstile, ...) while the measurements accrue over time (multiple temperature readings at regular interval from this sensor, multiple entries over time at that turnstile, ...)
Most of the time, these sensor readings from these type of IoT technologies do not contain location information. You will not find a cheap temperature sensors equipped with GPS-technology. It would be overkill anyway: explicit location information does not need to be embedded with each measurement of a stationary object.
This normalization of splitting up the spatial information of the object, from the metrics emitted by this object is a very common way to model data in “GIS” (“Geographic Information Systems”).
- The spatial information of the entities are captured in a “layer”. This spatial information is static. The documents have a primary-key field that uniquely identifies the real-world object. These can be points (e.g. for sensor-locations), or lines or polygons (for natural features). In Elasticsearch, this would be an index with a
geo_pointorgeo_shapefield, and another field (usually a keyword field) with the primary-key. - The other is a table that contains the metrics. This table has a foreign-key that references an entity in the layer. In Elasticsearch, this would be an index with a field (usually a keyword field) that can serve as the foreign-key.
The Maps app in Kibana allows you to link his “layer” and “table” using the Term Join functionality.
Example: mapping stationary sensor data
Since I don’t want to share the location of my actual house 
, let’s pretend I put up three temperature sensors in the White house. One in the oval office, the other on the entry-porch, and one in the rose-garden.
1. Create the layer
I used geojson.io to create a GeoJson-file with the three sensors.
The important bits here are the sensor_id primary-key field in the properties which uniquely identifies the sensor, and the location of the sensor
The file would look something like:
{
"type":"FeatureCollection",
"features":[
{
"type":"Feature",
"properties":{
"sensor_id":"dht22_1",
"location":"Oval office"
},
"geometry":{
"type":"Point",
"coordinates":[
-77.03710377216339,
38.897491797297405
]
}
},
{
"type":"Feature",
"properties":{
"sensor_id":"dht22_2",
"location":"Rose garden"
},
"geometry":{
"type":"Point",
"coordinates":[
-77.03740686178207,
38.89738742299468
]
}
},
{
"type":"Feature",
"properties":{
"sensor_id":"dht22_3",
"location":"Main entry"
},
"geometry":{
"type":"Point",
"coordinates":[
-77.03648686408997,
38.89756485921805
]
}
}
]
}
To ingest this file, we’ll use Kibana Maps. Create a new map, click add layer, and select GeoJson upload.
We can add some labels and colors in the styling editor, and voila, this is the reference layer.
Optimize this layer by turning off “apply global filter to layer data”. The locations of these sensors do not need to be filtered down based on any kind of filters in the global Kibana context.
2. Collect the temperature
We’ll skip over this step. Actually ingesting the sensor-readings into Elasticsearch is another topic altogether, and is very product dependent. You can check some IoT examples our here.
- Visualizing Room Climate Data with Conectric’s IoT Sensors
- Industrial internet of Things with the Elastic stack
The relevant bit here is that each document contains a foreign-key field that refers back to the sensor_id-field in the layer we created in step 1.
In out temperature index, the mapping should at least contain these fields:
{
"temperature":{
"mappings":{
"properties":{
"@timestamp":{
"type":"date"
},
"sensor_id":{
"type":"keyword"
},
"temperature":{
"type":"double"
}
}
}
}
}
Important here is to select keyword for the sensor_id field since it needs to function as a foreign key.
Also make sure to create an index-pattern over this index in Kibana.
3. Setup the term-join
Go back to the maps-app, and select that sensors-layer we created in step 1.
Setup the term-join:
The left-field and left-source is the sensor-layer with the spatial information. The right source and right-field is the temperature metrics collected by the sensors.
4. Specify the metric aggregation
For this term-join, we can compute any kind of metrics. Average, min, max temperature. Kibana is a perfect platform for time-based data, any Maps will automatically apply the global time-selection to the visualization.
5. Symbolize the sensors based on these values
Now we can adjust the symbology based on these values. I want to adjust the bubble size map average temperature and adjust the color based on average temperature size as well. Do this by selection “by value” styling.
And voila, here is our map:
Wrap-up
This post showed a client-side approach on how to solve this problem. You can also perform this "join", at ingest using the lookup-processor. This blog post shows an example on how to do that for private-IP addresses. Both approaches are valid.
The client-side approach shown here has no penalty as long as the number of entities in the "left" layer remains on the lower end (order of hundreds or thousands, at most). The size of the index with the metrics can obviously grow in the order of millions and above.
Hope this helped you getting started with mapping data that does not explicit location information. Using "Term joins" is a very generic approach, and is relevant for many use-cases. We used a local scale (single house), but the same technique works for global-scale as well (e.g. global data-center locations keyed by ip-ranges). It also works for spatial-data that is not point based, e.g. shopfloor plans where you represent a door as an actual line. For any question on this topic, discuss is the perfect venue.