# Issues filtering Elasticsearch logs in Docker with Modules

**URL:** <https://discuss.elastic.co/t/issues-filtering-elasticsearch-logs-in-docker-with-modules/263314>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [February 4, 2021, 11:19pm UTC](https://discuss.elastic.co/t/issues-filtering-elasticsearch-logs-in-docker-with-modules/263314 "2021-02-04T23:19:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![leosunmo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leosunmo/32/83452_2.png) [@leosunmo](https://discuss.elastic.co/u/leosunmo)\
**Post date:** [February 4, 2021, 11:19pm UTC](https://discuss.elastic.co/t/issues-filtering-elasticsearch-logs-in-docker-with-modules/263314/1 "2021-02-04T23:19:03Z")

</div>

Hey there,

I'm running in to some matching issues when I'm trying to use the Elasticsearch Module with the Docker provider.

I've followed [this blog post](https://xeraa.net/blog/2020_filebeat-modules-with-docker-kubernetes/) on how to make it play nicely together somewhat.

My issue is that it is not actually filtering on the log type in a predictable manner. I might be missing something obvious here and would love some pointers.

Here's my current test setup:

- es1: `docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --label co.elastic.logs/enabled=false --name es1 docker.elastic.co/elasticsearch/elasticsearch:7.10.2`
- es2: `docker run -e "discovery.type=single-node" --label co.elastic.logs/module=elasticsearch --name es2 docker.elastic.co/elasticsearch/elasticsearch:7.10.2`

es1 and es2, both in Docker on Linux, no docker-compose. es1 receives the logs and will have the ingest pipelines installed by Filebeat. es2 is the "sending" Elasticsearch with the module enabled through the Docker label.

I run Filebeat on my system, with the following config (more or less copied from above blog):

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/lib/docker/containers/${data.container.id}/*-json.log
        fields_under_root: true

      fields:
        type: elasticsearch
        cluster: prod

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-server-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '"type": "server"'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-deprecation-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '"type": "deprecation"'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-audit-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '"type": "audit"'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-slowlog-pipeline"
      when:
        and:
          - equals:
              container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"
          - contains:
              message: '_slowlog'
    - pipeline: "filebeat-%{[agent.version]}-elasticsearch-gc-pipeline"
      when.equals:
        container.labels.org_label-schema_url: "https://www.elastic.co/products/elasticsearch"

```

I run it with processor debug logs:

```auto
sudo ./filebeat -e -d 'processors'

```

Upon inspecting the debug log output I see mostly correctly identified logs (`"type\": \"server\"`) but occasionally I get things like this:

```auto
2021-02-05T11:51:11.046+1300	DEBUG	[processors]	processing/processors.go:203	Publish event: {
  "@timestamp": "2021-02-04T22:51:08.568Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.2",
    "pipeline": "filebeat-7.10.2-elasticsearch-audit-pipeline"
  },
  "stream": "stdout",
  "message": "{\"type\": \"server\", \"timestamp\": \"2021-02-04T22:51:08,565Z\", \"level\": \"INFO\", \"component\": \"o.e.n.Node\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"664c6c63325b\", \"message\": \"version[7.10.2], pid[6], build[default/docker/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.4.0-62-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]\" }",
  "event": {
    "module": "elasticsearch",
    "dataset": "elasticsearch.audit",
    "timezone": "+13:00"
  },
  "ecs": {
    "version": "1.5.0"
  },
  "service": {
    "type": "elasticsearch"
  },
  "input": {
    "type": "container"
  },
  "fileset": {
    "name": "audit"
  },
  "container": {
    "image": {
      "name": "docker.elastic.co/elasticsearch/elasticsearch:7.10.2"
    },
    "id": "664c6c63325bed09784e57117101cf4e9a766246364b25b30d0a158c91faf8ad",
    "name": "es2"
  },
}

```

I trimmed some stuff from the above, but as far as I can tell nothing should match the `message: '"type": "audit"'` rule here.

Any pointers or corrections super welcome!

---

<div class="post-metadata">

**Author:** ![leosunmo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leosunmo/32/83452_2.png) [@leosunmo](https://discuss.elastic.co/u/leosunmo)\
**Post date:** [February 18, 2021, 12:02am UTC](https://discuss.elastic.co/t/issues-filtering-elasticsearch-logs-in-docker-with-modules/263314/2 "2021-02-18T00:02:54Z")

</div>

I am still facing this issue. It seems FileBeats filtering logic is not sound somewhere. Could be a bug, but I'd like to confirm before I file one on Github.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 18, 2021, 2:03am UTC](https://discuss.elastic.co/t/issues-filtering-elasticsearch-logs-in-docker-with-modules/263314/3 "2021-03-18T02:03:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.