Hi all, I'm currently setting up Elasticsearch to handle continuous, real-time IoT sensor data (temperature, humidity, motion, etc.). Essentially, multiple devices send small packets of data every few seconds, and I aim to ensure that Elasticsearch can efficiently index this data without causing performance issues over time. I'm wondering if anyone here has experience optimizing similar IoT ingestion pipelines, particularly in terms of rollover strategies, mapping designs, or handling high write frequencies? Any insights about this? for reference: Real Life Examples of Internet of Things - The Engineering Projects
Hi @aria_234 Welcome to the community!.
The Elastic Stack can support large-scale IoT device telemetry ingestion and analytics.
Scale can mean different things to different people so you'll have to provide some examples...
Best advise any of us can give you is to do Proof of Concept POC with a mix of the type of devices. Understand how much each of them ingest and then we can interpolate what a large scale means.
Elastic supports many use cases with tens of thousands of devices....
What your exact architecture for that is can't be known until you do the testing.
There's other parts of a larger architecture such as durable queuing mechanism such as Kafka that often can add reliability, durability and queuing to support variable workloads.
To provide some additional details and perhaps we can help and or point you to additional materials.