Ingest Pipeline is not applying tag, what am I missing?

[jerrac (David Reagan)]

I've set up an ingest pipeline that sends my docker container logs through the appropriate built-in pipeline for the kind of logs the container is outputting. (See: Making Docker container logs go through the right beats module for processing? - #9 by stephenb )

I want to add a tag to the output of those pipelines so that I can adjust the searches backing the various dashboards to include logs from my containers. I thought all I had to do was add the tag to the processor's tag field. But that isn't working. There are no tags on the output.

What am I missing?

Here's the json for my pipeline:

[
  {
    "pipeline": {
      "name": "logs-apache.access-1.3.5",
      "if": "ctx.container.labels.com_docker_swarm_service_name.endsWith('-app') && ctx.stream == 'stdout'",
      "tag": "docker-apache-access",
      "ignore_failure": true
    }
  },
  {
    "pipeline": {
      "name": "logs-apache.error-1.3.5",
      "if": "ctx.container.labels.com_docker_swarm_service_name.endsWith('-app') && ctx.stream == 'stderr'",
      "tag": "docker-apache-error"
    }
  },
  {
    "pipeline": {
      "name": "logs-mysql.error-1.2.1",
      "if": "ctx.container.labels.com_docker_swarm_service_name.endsWith('-db') && ctx.stream == 'stderr'",
      "tag": "docker-mysql-error"
    }
  }
]

I was able to get the Apache Access logs search working by adding or (event.dataset:"docker" and http.*:*) to it, but that doesn't work for error logs. There are no Apache specific fields for error logs.

Thanks in advance!

[stephenb (Stephen Brown)]

So you will need to add additional processors to add fields or tag etc...

What I would probably do is change you "main routing" pipeline above to something to call another wrapper pipeline which will then call the actual module pipeline and then add fields / tags you want

This is a good pattern ... think modular code...

[
.....
  {
    "pipeline": {
      "name": "docker-apache-error",
      "if": "ctx.container.labels.com_docker_swarm_service_name.endsWith('-app') && ctx.stream == 'stderr'",
      "tag": "docker-apache-error"
    }
  },

Then create a pipeline that would be something like... calls the main pipeline then add fields / tags other logic custom for you

PUT _ingest/pipeline/docker-apache-error
{
  "description": "docker apache pipeline wrapper",
  "processors": [
   {
    "pipeline": {
      "name": "logs-apache.error-1.3.5",
      "tag": "docker-apache-error"
    },
    {
      "set": {
        "field": "event.dataset,
        "value": "apache.error"
      }
    }
  ]
}