Filebeat not forwarding tags to logstash

Here's my filebeat config:

logging.level: debug
filebeat.registry_file: /tmp/registry
 filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /log/*
 shipper:
  tags:
    - source:docker
    - pod:sts-emergency-dev-main-682579278-809qq
    - k8s_node_name:ldm-0105029
    - pod_ip:10.42.77.3
 output.logstash:
  hosts:
    - logstashdev:5000
 output.console:
  pretty: true
 logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/filebeat

My problem is that tags don't seem to get forwarded into logstash. This is a dev environment running v5.5 of both logstash and filebeats - we have a similar setup in prod on 2.4.1 (tho with different tag names), and we haven't seen this problem there.

So I guess my question is: Did something change between 2.4.x and 5.5.x in how tags are handled? We rely on them heavily for filtering data in ES, so we kinda need them :expressionless:

Here's output from logstash parsing the beat input. As you can see, only the standard "beats" tag is sent in, and nothing else:

{
               "remote_addr" => "10.42.127.34",
                    "offset" => 1783,
             "upstream_addr" => "127.0.0.1:5000",
           "body_bytes_sent" => 0,
                "input_type" => "log",
            "request_method" => "GET",
                    "source" => "/log/access.log",
                      "type" => "log",
                       "uri" => "/health",
               "request_uri" => "/health",
           "http_user_agent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36",
               "host_header" => "sts-emergency-main.dev.rikstv.no",
                      "tags" => [
        [0] "beats_input_codec_json_applied"
    ],
               "remote_user" => "",
           "upstream_status" => 200,
                "@timestamp" => 2017-08-14T23:09:42.511Z,
              "request_time" => 0.002,
      "http_x_forwarded_for" => "10.245.11.233, 10.42.0.205",
                  "@version" => "1",
                      "beat" => {
        "hostname" => "sts-emergency-dev-main-682579278-mjdn9",
            "name" => "sts-emergency-dev-main-682579278-mjdn9",
         "version" => "5.5.1"
    },
                      "host" => "sts-emergency-dev-main-682579278-mjdn9",
             "http_referrer" => "",
    "upstream_response_time" => 0.002,
                    "status" => 200
}

I've also tried quoting each tag "item" in double quotes, just to make sure it's valid yaml, without it seeming to have any effect

The trick turned out the be moving the tag attribute to the prospector, instead of keeping it in the shipper. All good now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.