I would like to report on the experience of developing a simple home automation data collection and processing pipeline able to work with data and metrics from Domoticz and from many other various IoT devices and scripts.
In this kind of environment, the wide variety of log and metrics formats may look like a nightmare for automation, but actually for heterogeneous IoT and home automation data, combining Fluent Bit with Elasticsearch ingest pipelines provides a practical way of progressively converting unstructured log streams into structured data by dividing the data extraction process into two steps, the first one using Fluent-Bit's parsers and filters, and the second one using Elasticsearch's ingest pipelines and accurate mappings.
With this approach, the various steps of data extraction and formatting is made easy and flexible. The key design principle is to avoid making any single component responsible for the complete transformation.
The resulting architecture is small enough for a home automation environment while remaining flexible enough to accommodate additional sensors, scripts and data sources.
The complete configuration is described and available on GitHub (repository cchaussat/fluent-bit-elasticsearch-iot-logging).
I am interested in getting feedback, comments and questions, as well as in reading on similar or complementary or different experience from the community on this topic.