High volume apache httpd 2.4 access log processing


(Davidbernard) #1

I am wanting to move from "scribe" to ElasticStack for processing apache httpd 2.4 access logs.

I have identified the following approaches:

Option 1

CustomLog with pipe configuration for FileBeat processing stdin.

Option 2

CustomLog with pipe configuration to "rotatelog". FileBeat processing on disk logs

Option 3

Standard logging to file FileBeat processing on disk logs.

Typically we are processing 4,000 transactions per second.

Does anyone have experience with these options and transaction rates they could share?


(ruflin) #2

I would recommend you to go with option 3 as this makes it possible to use disk also as a buffer in case of peaks, when needing to upgrade FB etc.

4k should definitively work with FB but I would recommend to check out the throughput on your setup as it not only depends on FB but also size of your ES cluster, network etc.


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.