Filebeat not forwarding tags to logstash

Elastic Stack Beats
1

Here's my filebeat config:

logging.level: debug
filebeat.registry_file: /tmp/registry
 filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /log/*
 shipper:
  tags:
    - source:docker
    - pod:sts-emergency-dev-main-682579278-809qq
    - k8s_node_name:ldm-0105029
    - pod_ip:10.42.77.3
 output.logstash:
  hosts:
    - logstashdev:5000
 output.console:
  pretty: true
 logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/filebeat

My problem is that tags don't seem to get forwarded into logstash. This is a dev environment running v5.5 of both logstash and filebeats - we have a similar setup in prod on 2.4.1 (tho with different tag names), and we haven't seen this problem there.

So I guess my question is: Did something change between 2.4.x and 5.5.x in how tags are handled? We rely on them heavily for filtering data in ES, so we kinda need them :expressionless:

Here's output from logstash parsing the beat input. As you can see, only the standard "beats" tag is sent in, and nothing else:

{
               "remote_addr" => "10.42.127.34",
                    "offset" => 1783,
             "upstream_addr" => "127.0.0.1:5000",
           "body_bytes_sent" => 0,
                "input_type" => "log",
            "request_method" => "GET",
                    "source" => "/log/access.log",
                      "type" => "log",
                       "uri" => "/health",
               "request_uri" => "/health",
           "http_user_agent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36",
               "host_header" => "sts-emergency-main.dev.rikstv.no",
                      "tags" => [
        [0] "beats_input_codec_json_applied"
    ],
               "remote_user" => "",
           "upstream_status" => 200,
                "@timestamp" => 2017-08-14T23:09:42.511Z,
              "request_time" => 0.002,
      "http_x_forwarded_for" => "10.245.11.233, 10.42.0.205",
                  "@version" => "1",
                      "beat" => {
        "hostname" => "sts-emergency-dev-main-682579278-mjdn9",
            "name" => "sts-emergency-dev-main-682579278-mjdn9",
         "version" => "5.5.1"
    },
                      "host" => "sts-emergency-dev-main-682579278-mjdn9",
             "http_referrer" => "",
    "upstream_response_time" => 0.002,
                    "status" => 200
}
2

I've also tried quoting each tag "item" in double quotes, just to make sure it's valid yaml, without it seeming to have any effect

3

The trick turned out the be moving the tag attribute to the prospector, instead of keeping it in the shipper. All good now.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.