High volume apache httpd 2.4 access log processing

Elastic Stack Beats
1

I am wanting to move from "scribe" to ElasticStack for processing apache httpd 2.4 access logs.

I have identified the following approaches:

Option 1

CustomLog with pipe configuration for FileBeat processing stdin.

Option 2

CustomLog with pipe configuration to "rotatelog". FileBeat processing on disk logs

Option 3

Standard logging to file FileBeat processing on disk logs.

Typically we are processing 4,000 transactions per second.

Does anyone have experience with these options and transaction rates they could share?

2

I would recommend you to go with option 3 as this makes it possible to use disk also as a buffer in case of peaks, when needing to upgrade FB etc.

4k should definitively work with FB but I would recommend to check out the throughput on your setup as it not only depends on FB but also size of your ES cluster, network etc.

1 Like
3

This topic was automatically closed after 21 days. New replies are no longer allowed.