Error and access logs are merged: Filebeat + Elasticsearch + php-fpm on Docker

Elastic Stack Beats
1

I'm having an annoying problem with the way php-fpm behaves on Docker; it writes both error and access logs to stderr! This is done because php-fpm closes the stdout stream when it starts up, so it's impossible to write there.

Is there a way with Filebeat modules to re-classify "error" log entries coming from stderr with a certain Grok pattern, to become proper "access" log entries when being ingested into Elasticsearch?

I do not use Logstash, only Filebeat + Elasticsearch.

2

Hi @remimikalsen and welcome :slight_smile:

You can use an ingest pipeline to parse and filter any kind of logs, take a look to the documentation to see how to use ingest pipelines with filebeat: https://www.elastic.co/guide/en/beats/filebeat/7.0/configuring-ingest-node.html

Ingest pipelines include a grok processor.

3

Thank you for the reply @jsoriano. I'll look into it; I'd probably be able to solve all my metadata issues based on conditional processors, if I also tweak the error pipeline in the Filebeat Apache module in order for the error pipeline to parse access logs.

But after thinking more about the topic, I'm wondering if I shouldn't make a new module instead. After all, I'm using the Apache module for php-fpm just because the log patterns correspond more or less.

Another alternative is to create a new module called "PHP-FPM" based on vanilla Apache. This module would have only one pipeline reading from stderr to a fileset called fpmlog or something. That one pipeline would incorporate the Grok patterns from both error and access in vanilla Apache. That way I would get the metadata right for my php-fpm container.

It leaves me with another problem though, and it's the fact that my Nginx proxies will still ship the logs coming from the upstream php-fpm servers as errors. But then I can create a custom Nginx module for php-fpm upstream containers.

Does it seem a reasonable solution to create new Filebeat modules to solve this issue?

4

Having a module for php-fpm would be definitively great if it has some difference with its log, but if they use the same format as apache it'd be better if we could use the apache one as is.
Also it'd be preferable to don't mix different kinds of formats in the same pipeline if possible. If for this specific scenario you need a custom solution I think it'd be better to do it with a custom pipeline.

We can give a second try to send access and error logs to different streams, this should be possible. How are you trying to configure php fpm for this?

Do you mean that nginx is running in the same container as php fpm and it is also sending its logs to stderr? This will make an even stronger argument in favour of a custom pipeline instead of a module for your case.
Or you can also consider running nginx and php-fpm in different containers. They could be run in the same network namespace so Nginx can make local connections, or they could share a volume with the unix socket if you are using them.

5

Unfortunately, it's not my choice to put access logs in stderr. The problem is the official php-fpm image that shuts down stdout completely (I have no idea why, but there must be som weighty argument supporting it). That's why the guys behind Nextcloud and others ship images based on php-fpm with php-fpm.conf like this:

error_log = /proc/self/fd/2
...
; if we send this to /proc/self/fd/1, it never appears
access.log = /proc/self/fd/2

In my setup, Nginx is in a different container. It serves the static content directly and proxies any PHP-scripts to the php-fpm container. I may be wrong about Nginx relaying logs - the two containers should log completely separately I guess - but I'll have to fix the first issue in order to clean up the logs properly and get a better overview :slight_smile:

6

Umm, if it is a so general problem when using phpfpm in docker then maybe it worths to have a module for it, would you like to open a PR for that?

If Nginx is in a different container it should be possible to use the nginx module for its logs.

7

I'd be more than happy to contribute the module when or if I get it working reasonably. At that point I'll have to get some guidance to contribute it properly.

8

Thanks! You can find a guide to create new filebeat modules in the documentation: https://www.elastic.co/guide/en/beats/devguide/7.0/filebeat-modules-devguide.html

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.