Filebeat not dropping the entire events/log

Elastic Stack Beats
,
1

I don't want to collect logs from all containers, so I want to discard a few container logs.
In the below case, I don't want to capture logs from logstash and elastic.

Actual:
In the logs, I see only the "container" field has been dropped from the JSON body nothing else.

Expected:
Drop the entire JSON message.

filebeat.inputs:
  - type: docker
    combine_partial: true
    containers:
      path: "/usr/share/dockerlogs/data"
      stream: "stdout"
      ids:
        - "*"
    exclude_files: ['\.gz$']
    ignore_older: 10m
    processors:
      drop_event.when:
        or:
        - equals:
            docker.container.name: "s_logstash_1"
        - equals:
            docker.container.name: "s_es_1"

Output:

"container" field is missing.

3e[36mlogstash_1 |e[0m {
e[36mlogstash_1 |e[0m "host" => {
e[36mlogstash_1 |e[0m "name" => "5e15071a0012"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "@version" => "1",
e[36mlogstash_1 |e[0m "tags" => [
e[36mlogstash_1 |e[0m [0] "beats_input_codec_plain_applied"
e[36mlogstash_1 |e[0m ],
e[36mlogstash_1 |e[0m "@timestamp" => 2020-08-30T10:01:53.150Z,
e[36mlogstash_1 |e[0m "agent" => {
e[36mlogstash_1 |e[0m "version" => "7.7.1",
e[36mlogstash_1 |e[0m "id" => "da6fda98-f680-43a6-8f38-fc80d0ff661e",
e[36mlogstash_1 |e[0m "type" => "filebeat",
e[36mlogstash_1 |e[0m "ephemeral_id" => "0120ef2f-2ceb-4a8a-b9aa-328dc7a203ee",
e[36mlogstash_1 |e[0m "hostname" => "5e15071a0012"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "message" => "{"type": "server", "timestamp": "2020-08-30T10:01:53,150Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "elasticsearch", "node.name": "08ebabd41a9d", "message": "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "cluster.uuid": "09eHr0q7Say8F-1lWM50Lg", "node.id": "d2R6IIMhSW-cF6w8U00nqA" }",
e[36mlogstash_1 |e[0m "input" => {
e[36mlogstash_1 |e[0m "type" => "docker"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "stream" => "stdout",
e[36mlogstash_1 |e[0m "ecs" => {
e[36mlogstash_1 |e[0m "version" => "1.5.0"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "log" => {
e[36mlogstash_1 |e[0m "file" => {
e[36mlogstash_1 |e[0m "path" => "/usr/share/dockerlogs/data/08ebabd41a9dbf62295cd207365200ff617a371df5427b0857bcf82298bbed52/08ebabd41a9dbf62295cd207365200ff617a371df5427b0857bcf82298bbed52-json.log"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "offset" => 25994
e[36mlogstash_1 |e[0m }
e[36mlogstash_1 |e[0m }
e[36mlogstash_1 |e[0m {
e[36mlogstash_1 |e[0m "host" => {
e[36mlogstash_1 |e[0m "name" => "5e15071a0012"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "@version" => "1",
e[36mlogstash_1 |e[0m "tags" => [
e[36mlogstash_1 |e[0m [0] "beats_input_codec_plain_applied"
e[36mlogstash_1 |e[0m ],
e[36mlogstash_1 |e[0m "@timestamp" => 2020-08-30T10:01:53.796Z,
e[36mlogstash_1 |e[0m "agent" => {
e[36mlogstash_1 |e[0m "version" => "7.7.1",
e[36mlogstash_1 |e[0m "id" => "da6fda98-f680-43a6-8f38-fc80d0ff661e",
e[36mlogstash_1 |e[0m "type" => "filebeat",
e[36mlogstash_1 |e[0m "ephemeral_id" => "0120ef2f-2ceb-4a8a-b9aa-328dc7a203ee",
e[36mlogstash_1 |e[0m "hostname" => "5e15071a0012"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "message" => "{"type": "server", "timestamp": "2020-08-30T10:01:53,796Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "elasticsearch", "node.name": "08ebabd41a9d", "message": "license [ec606c74-b286-4354-be8e-229fc46425af] mode [basic] - valid", "cluster.uuid": "09eHr0q7Say8F-1lWM50Lg", "node.id": "d2R6IIMhSW-cF6w8U00nqA" }",
e[36mlogstash_1 |e[0m "input" => {
e[36mlogstash_1 |e[0m "type" => "docker"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "stream" => "stdout",
e[36mlogstash_1 |e[0m "ecs" => {
e[36mlogstash_1 |e[0m "version" => "1.5.0"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "log" => {
e[36mlogstash_1 |e[0m "file" => {
e[36mlogstash_1 |e[0m "path" => "/usr/share/dockerlogs/data/08ebabd41a9dbf62295cd207365200ff617a371df5427b0857bcf82298bbed52/08ebabd41a9dbf62295cd207365200ff617a371df5427b0857bcf82298bbed52-json.log"
e[36mlogstash_1 |e[0m },
e[36mlogstash_1 |e[0m "offset" => 26466
e[36mlogstash_1 |e[0m }
e[36mlogstash_1 |e[0m }

2

Hi!

This is quite weird! Could you share more info about the version you are using?

Also do you see anything suspicious in the logs regarding the drop_event processor? The processor is supposed to just drop events when conditions are met. Could you verify that processor does not fail for some reason?

C.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.