Unable add a field (grok) with docker_metadata as its value

[yanivomc1 (Yanivomc)]

Hi Guys,
spent a day on this thing.

i got a json log that being sent to filebeat and from there to logstash.
The problem:
i've added docker_metadata in filebeat,
i also see the docker metadata there in logstash ouput but im unable to add a custom field that has a vaule of one of the docker_metadata..

those are the lables in my dockers;
<
"Labels": {
"com.amazonaws.ecs.cluster": "ecsFrontendCluster",
"com.amazonaws.ecs.container-name": "site",
"com.amazonaws.ecs.task-arn": "xxxxxxx-41d4-a410-xx",
"com.amazonaws.ecs.task-definition-family": "site",
"com.amazonaws.ecs.task-definition-version": "150"
}
/>

the log file that being ingested by logstash (ouput) -

<
......
......
"docker": {
"container": {
"labels": {
"com": {
"amazonaws": {
"ecs": {
"task-definition-version": "150",
"task-arn": "xxxxxx",
"cluster": "ecsFrontendCluster",
"container-name": "site",
"task-definition-family": "site"
}
}
}
}
.....
/>

what i wish to do is the following:
i want to have a new field that will be under my metadata:
<
filter {
mutate {
copy => { "[docker][container][Labels][com][amazonaws][ecs][task-definition-family]" => "[@metadata][docker_role]]" }
}
}
/>

then i wish to use the metadata as an index name

whatever i did,
i wasn't able to get access to the docker values.

thanks!

[Christian_Dahlqvist]

It looks like you are specifying “Labels” instead of “labels” in your mutate filter.

[yanivomc1 (Yanivomc)]

Hi thanks you for your answer.
i wish it was that -
The "Labels" instead of "labels" was a miss when i wrote the post here.

i'm using "labels" but it's not working.

filter {
  mutate {
        copy  => { "[docker][container][labels][com][amazonaws][ecs][task-definition-version]" => "[@metadata][docker_role]" }
  }
}

from debug log:
[2018-11-04T07:26:53,615][DEBUG][logstash.filters.mutate ] config LogStash::Filters::Mutate/@copy = {"[docker][container][labels][com][amazonaws][ecs][task-definition-version]"=>"[@metadata][docker_role]"}

is it something that suppose to work ?
extracting those fields i mean.

[Christian_Dahlqvist]

Do you have those fields in the event?

[yanivomc1 (Yanivomc)]

Here is my simple filter:
filter {
mutate {
add_field => { "[@metadata][docker_role]" => "" }
copy => { "[docker][container][labels][com][amazonaws][ecs][task-definition-version]" => "[@metadata][docker_role]" }
}
}
here is the complete log:

{
          "path" => "/logs/log.json",
    "@timestamp" => 2018-11-04T07:34:32.693Z,
        "_index" => "k1-[docker][container][labels][com][amazonaws][ecs][container-name]-2018",
     "@metadata" => {
               "host" => "333333333",
        "docker_role" => "",
               "path" => "/logs/log.json"
    },
          "host" => "333333333",
         "_type" => "doc",
      "@version" => "1",
       "_source" => {
            "offset" => 3333,
             "level" => 100,
           "channel" => "staging",
           "ecsinfo" => {
            "task-definition-version" => "[docker][container][labels][com][amazonaws][ecs][task-definition-version]"
        },
        "prospector" => {
            "type" => "docker"
        },
            "source" => "/var/lib/docker/containers/e4349b8c575c1bd88eddeaf1a4a067f91cc11f3d55210c0e8693397b198778cb/e4349b8c575c1bd88eddeaf1a4a067f91cc11f3d55210c0e8693397b198778cb-json.log",
           "message" => "xxxxxxxx: ",
            "docker" => {
            "container" => {
                 "image" => "xxxxxxxx:1757",
                  "name" => "xxxxxxxx",
                    "id" => "xxxxxxxx",
                "labels" => {
                    "com" => {
                        "amazonaws" => {
                            "ecs" => {
                                                "cluster" => "ecsFrontendCluster",
                                "task-definition-version" => "150",
                                 "task-definition-family" => "site",
                                         "container-name" => "site",
                                               "task-arn" => "xxxxxxxx"
                            }
                        }
                    }
                }
            }
        },
              "tags" => [
            [0] "beats_input_codec_json_applied"
        ],
             "input" => {
            "type" => "docker"
        },
          "datetime" => {
                     "date" => "2018-11-03 21:54:28.068575",
                 "timezone" => "UTC",
            "timezone_type" => 3
        },
        "level_name" => "DEBUG",
        "@timestamp" => "2018-11-03T21:54:28.358Z",
            "stream" => "stdout",
             "extra" => {
                "referrer" => nil,
                  "server" => "_",
                     "uid" => "xxxxxxxx",
             "http_method" => "GET",
                      "ip" => "xxxxxxxx",
            "memory_usage" => "xxxxxxxx",
                     "url" => "xxxxxxxx"
        },
              "host" => {
            "name" => "xxxxxxxx"
        },
              "beat" => {
            "hostname" => "xxxxxxxx",
                "name" => "xxxxxxxx",
             "version" => "6.3.2"
        },
           "context" => [],
          "@version" => "1"
    },
           "_id" => "xxxxxxxx",
          "type" => "json",
        "_score" => 1,
        "fields" => {
        "@timestamp" => [
            [0] "2018-11-03T21:54:28.358Z"
        ]
    }
}