# Best queue for devices that are mostly offline **URL:** \ **Category:** Elasticsearch\ **Created:** [March 31, 2020, 10:41am UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835 "2020-03-31T10:41:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dreamspy](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@dreamspy](https://discuss.elastic.co/u/dreamspy)\ **Post date:** [March 31, 2020, 10:41am UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835/1 "2020-03-31T10:41:20Z")
I'm wondering what would be the best queue option for computers that are mostly offline. Logstash, Redis, Kafka, RabbitMQ? I need to pump logs/metrics from those devices to the ES cloud. But the devices are online maybe just 1-2 hours a day, so I need a very robust queue solution. P.s. this is a repost of an earlier post that didn't receive any answers, most likely because it was way to long and detailed: > [@Use Logstash persistent queue, or use Redis/Kafka/RabbitMQ? (On a server that's offline most of the time)](https://discuss.elastic.co/t/use-logstash-persistent-queue-or-use-redis-kafka-rabbitmq-on-a-server-thats-offline-most-of-the-time/225502/2): > > I would like to add that the data that'll be stored in the DB on the robot itself is only a small subset of the whole data that'll be stored on the AWS server. So the idea of writing only to the local database, and then using something to sync that DB to the AWS one, would not work in this case. regards Frimann ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 31, 2020, 11:41am UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835/2 "2020-03-31T11:41:41Z")
If the devices are only occasionally online you need to queue things on the devices themselves, rather than using a fancy external distributed queue. Why not treat the log file itself as the queue? I think [Filebeat](https://www.elastic.co/beats/filebeat) will deal with this ok since it should handle sporadic connections to Elasticsearch. For metrics you could try Metricbeat with its [file spool queue](https://www.elastic.co/guide/en/beats/metricbeat/current/configuring-internal-queue.html#configuration-internal-queue-spool). ---
**Author:** ![dreamspy](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@dreamspy](https://discuss.elastic.co/u/dreamspy)\ **Post date:** [March 31, 2020, 4:26pm UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835/3 "2020-03-31T16:26:28Z")
That sounds like a simple solution. One question that comes into mind. I would probably need to clear out the log files (oldest entries) periodically, should Filebeat handle this without problems? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 31, 2020, 5:23pm UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835/4 "2020-03-31T17:23:07Z")
You'll need to organise some kind of log rotation; I'm pretty sure that Filebeat knows how to handle rotating logs but may need special care if the logs rotate multiple times while the node is offline. Probably best to ask about that on the Beats forum. Alternatively, Filebeat also has a file spool queue which should do the right thing even with rotating logs. ---
**Author:** ![dreamspy](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@dreamspy](https://discuss.elastic.co/u/dreamspy)\ **Post date:** [March 31, 2020, 10:36pm UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835/5 "2020-03-31T22:36:45Z")
The logs will be rotated much less than the data is offloaded. But I'll check in the Beats forum if this starts to be a problem. Thanks for the help 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 28, 2020, 10:36pm UTC](https://discuss.elastic.co/t/best-queue-for-devices-that-are-mostly-offline/225835/6 "2020-04-28T22:36:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.