Multiple modules and/or pipelines in containers in kubernetes autodiscover with hints

[arpderp]

Hi,

I have containers running drupal and drupal is among another things generating its own syslog messages and nginx access logs.

I want to capture nginx access logs and the drupal syslogs and pass them to elasticsearch and nginx should be parased by the built in nginx module in filebeat and the drupal syslogs should be parsed by my custom pipeline drupal-syslog.

But I can't get it to work like I want it.

What I have done is testing the config for each of the one at a time to check that they actually work.

annotaions with nginx access only

co.elastic.logs/enabled : true
co.elastic.logs/fileset.stdout : access
co.elastic.logs/include_lines : ^(\b25[0-5]|\b2[0-4][0-9]|\b[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}
co.elastic.logs/module : nginx

As expected I get only the access logs and the log entries gets passed through with the nginx module. event.module : nginx etc. is present in the documents.

annotaions with drupal syslog only

co.elastic.logs/enabled : true
co.elastic.logs/include_lines : ^\[SYSLOG\] drupal
co.elastic.logs/pipeline : drupal-syslog

As expected I get only the drupal syslogs and the log entries get passed through the drupal-syslog pipeline. I get drupal.fields : some_data based on how I configured the pipeline.

So both configurations work as they should!

However I want to combine these two.

annotaions with both drupal syslog and nginx

co.elastic.logs/enabled : true
co.elastic.logs/fileset.stdout : access
co.elastic.logs/include_lines : ^(\b25[0-5]|\b2[0-4][0-9]|\b[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}
co.elastic.logs/module : nginx

co.elastic.logs/1.enabled : true
co.elastic.logs/1.include_lines : ^\[SYSLOG\] drupal
co.elastic.logs/1.pipeline : drupal-syslog

Here both log entries gets picked up but the issue is that they both go through the nginx module.
Documents created with the drupal syslog entries gets tagged with event.module : nginx etc, but can't be parsed since the message doesn't fit. They don't get parsed by the drupal-syslog pipeline at all. Nginx access entries is parsed correctly.

If I set drupal syslog to be without 1. and nginx with 1. the issue is reversed. drupal syslog gets parsed correctly and nginx access entry pops up as an entry in the message field.
Basically the issue is reversed.

I've also tried ordering them with 0. and 1. also with 1. and 2. without any luck. Here they don't work at all. Probably as intended^^

I've based this on Multiple sets of hints

I'm using Elasticsearch and Kibana with version 8.5.1 and Filebeat version 8.5.1.

I've also tested with Elasticsearch with version 8.4.3 and Filebeat version 7.17.3.

filebeat.yml

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        enabled: false
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log