Sequence Number Generation

Elastic Stack Logstash
1

We would like to annotate our inbound log messages with sequence numbers which we can then use to detect and measure message loss.

What is the best way to go about this? Use in-line ruby code to generate sequence numbers or an external generator, like redis? And might an external generator have a noticeable impact on throughput?

Thx
D

2

Has noone crossed this particular bridge before?

3

If you use ruby then you will need to make sure that pipeline.workers is set to 1 and pipeline.ordered evaluates to true. That limits throughput.

I cannot speak to solutions external to logstash.

4

If you want to measure message losses, you can use the pipeline statistic. Have you tried monitoring Logstash or ELK stack with Metricbeat or ElasticAgent?