Best queue for devices that are mostly offline

Elastic Stack Elasticsearch
1

I'm wondering what would be the best queue option for computers that are mostly offline. Logstash, Redis, Kafka, RabbitMQ?

I need to pump logs/metrics from those devices to the ES cloud. But the devices are online maybe just 1-2 hours a day, so I need a very robust queue solution.

P.s. this is a repost of an earlier post that didn't receive any answers, most likely because it was way to long and detailed:

regards
Frimann

2

If the devices are only occasionally online you need to queue things on the devices themselves, rather than using a fancy external distributed queue. Why not treat the log file itself as the queue? I think Filebeat will deal with this ok since it should handle sporadic connections to Elasticsearch.

For metrics you could try Metricbeat with its file spool queue.

1 Like
3

That sounds like a simple solution.

One question that comes into mind. I would probably need to clear out the log files (oldest entries) periodically, should Filebeat handle this without problems?

4

You'll need to organise some kind of log rotation; I'm pretty sure that Filebeat knows how to handle rotating logs but may need special care if the logs rotate multiple times while the node is offline. Probably best to ask about that on the Beats forum.

Alternatively, Filebeat also has a file spool queue which should do the right thing even with rotating logs.

5

The logs will be rotated much less than the data is offloaded. But I'll check in the Beats forum if this starts to be a problem.

Thanks for the help :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.