Issues filtering Elasticsearch logs in Docker with Modules

Elastic Stack Beats
,
1

Hey there,

I'm running in to some matching issues when I'm trying to use the Elasticsearch Module with the Docker provider.

I've followed this blog post on how to make it play nicely together somewhat.

My issue is that it is not actually filtering on the log type in a predictable manner. I might be missing something obvious here and would love some pointers.

Here's my current test setup:

  • es1: docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --label co.elastic.logs/enabled=false --name es1 docker.elastic.co/elasticsearch/elasticsearch:7.10.2
  • es2: docker run -e "discovery.type=single-node" --label co.elastic.logs/module=elasticsearch --name es2 docker.elastic.co/elasticsearch/elasticsearch:7.10.2

es1 and es2, both in Docker on Linux, no docker-compose. es1 receives the logs and will have the ingest pipelines installed by Filebeat. es2 is the "sending" Elasticsearch with the module enabled through the Docker label.

I run Filebeat on my system, with the following config (more or less copied from above blog):

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/lib/docker/containers/${data.container.id}/*-json.log
        fields_under_root: true

      fields:
        type: elasticsearch
        cluster: prod


output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-server-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '"type": "server"'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-deprecation-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '"type": "deprecation"'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-audit-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '"type": "audit"'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-slowlog-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '_slowlog'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-gc-pipeline"
      when.equals:
        container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"

I run it with processor debug logs:

sudo ./filebeat -e -d 'processors'

Upon inspecting the debug log output I see mostly correctly identified logs ("type\": \"server\") but occasionally I get things like this:

2021-02-05T11:51:11.046+1300	DEBUG	[processors]	processing/processors.go:203	Publish event: {
  "@timestamp": "2021-02-04T22:51:08.568Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.2",
    "pipeline": "filebeat-7.10.2-elasticsearch-audit-pipeline"
  },
  "stream": "stdout",
  "message": "{\"type\": \"server\", \"timestamp\": \"2021-02-04T22:51:08,565Z\", \"level\": \"INFO\", \"component\": \"o.e.n.Node\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"664c6c63325b\", \"message\": \"version[7.10.2], pid[6], build[default/docker/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.4.0-62-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]\" }",
  "event": {
    "module": "elasticsearch",
    "dataset": "elasticsearch.audit",
    "timezone": "+13:00"
  },
  "ecs": {
    "version": "1.5.0"
  },
  "service": {
    "type": "elasticsearch"
  },
  "input": {
    "type": "container"
  },
  "fileset": {
    "name": "audit"
  },
  "container": {
    "image": {
      "name": "docker.elastic.co/elasticsearch/elasticsearch:7.10.2"
    },
    "id": "664c6c63325bed09784e57117101cf4e9a766246364b25b30d0a158c91faf8ad",
    "name": "es2"
  },
}

I trimmed some stuff from the above, but as far as I can tell nothing should match the message: '"type": "audit"' rule here.

Any pointers or corrections super welcome!

2

I am still facing this issue. It seems FileBeats filtering logic is not sound somewhere. Could be a bug, but I'd like to confirm before I file one on Github.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.