We've been using Elastic in capturing and consolidating IoT data however - my experience with Elastic is however limited to about 6 months.
Consider the scenario:
- large number of IoT devices with known serial numbers are sending timestamped data
- a separate index contains the
geo_pointfor each device (key being the unique serial number)
- data/messages from the devices are enriched with the matching
geo_point(again, key being the unique serial number)
This is great for creating maps (especially heatmaps).
However is this the best way to do it? Is enriching each and every message with the geo_point info not wasteful by taking up a lot of space that could be correlated from another index. The expected number of messages from the IoT devices is increasing rapidly (ex: 6 months ago the average was 1M/week, now is at 5M/week).
Any suggestions on how to improve or optimize this would be much appreciated.