Index name based on docker-compose service name

[fulkon (Sergey)]

Is there any way to set index name based on docker-compose service name?
Corresponding name is in container.labels.com_docker_compose_service field.
Tried this in input:

    - type: container
      paths:
        - '/var/lib/docker/containers/*/*.log'
      processors:
        - add_docker_metadata:
            host: "unix:///var/run/docker.sock"

        - decode_json_fields:
            fields: ["message"]
            target: "json"
            overwrite_keys: true
      fields:
        application: "%{[container.labels.com_docker_compose_service]}"

with this:

  output:
    kafka:
      topic: "%{[fields.application]}"

but got nothing in topic.
Please, help

[fulkon (Sergey)]

Update:
Tried next in filebeat:>

 - type: log
   paths:
     - '/var/lib/docker/containers/*/*.log'
   processors:
     - add_docker_metadata:
         host: "unix:///var/run/docker.sock"
         labels.dedot: false
   index: "%{[@metadata][container.labels.com_docker_compose_service]}-%{+yyyy.MM.dd}"

Deleted split by topics in output.
Got:

"_index": "null-2020.12.14",

and

"tags": [
      "_grokparsefailure"
    ],

though I haven't any grok filters in filebeat or logstash

[mtojek (Marcin Tojek)]

Could you post a sample document that has been ingested?

[fulkon (Sergey)]

{
  "_index": "null-2020.12.14",
  "_type": "_doc",
  "_id": "G3JBYHYBI1N_HhbJPFe6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "message": "{<escaped_json_here>}",
    "input": {
      "type": "log"
    },
    "log": {
      "offset": 82324,
      "file": {
        "path": "/var/lib/docker/containers/b3591a3f329074e2a6d7a971687dc4444b24b2247fd294a1d5a2a2507eca6935/b3591a3f329074e2a6d7a971687dc4444b24b2247fd294a1d5a2a2507eca6935-json.log"
      }
    },
    "container": {
      "image": {
        "name": "<private_registry_url>:9398b666d3e14cfd483bd6dddbb7fa46f3895a64"
      },
      "labels": {
        "com_docker_compose_project": "sandbox-07",
        "com_docker_compose_project_config_files": "map.yaml",
        "com_docker_compose_container-number": "1",
        "com_docker_compose_version": "1.26.0",
        "com_docker_compose_oneoff": "False",
        "com_docker_compose_config-hash": "bdfc017d58f781ac2a77eed5cc353a10dc15fcf4054ae5c88ff54ba9697addfe",
        "com_docker_compose_service": "map",
        "com_docker_compose_project_working_dir": "/opt/app/sandbox-07"
      },
      "id": "b3591a3f329074e2a6d7a971687dc4444b24b2247fd294a1d5a2a2507eca6935",
      "name": "map_sandbox-07"
    },
    "ecs": {
      "version": "1.6.0"
    },
    "agent": {
      "version": "7.10.0",
      "id": "2595843c-227d-474d-b087-dbd02ee5095b",
      "hostname": "<hostname>",
      "name": "<hostname>",
      "ephemeral_id": "5d6d8eff-631d-4fb9-8906-1730ef4b7104",
      "type": "filebeat"
    },
    "host": {
      "name": "<hostname>"
    },
    "tags": [
      "_grokparsefailure"
    ],
    "@timestamp": "2020-12-14T07:57:35.681Z"
  },
  "fields": {
    "@timestamp": [
      "2020-12-14T07:57:35.681Z"
    ]
  },
  "sort": [
    1607932655681
  ]
}