Pod annotation co.elastic.logs/pipeline does not work?

[Kosodrom (Kosmodrom)]

Hi,

I have a Logstash pipeline pipeline running in kubernetes. I am collecting all logs from the Kubernetes cluster using filebeat with activated hit based autodiscover. I would like to force filebeat to use a specific ingest pipeline for my Logstash pipeline logs.

In order to achieve this I have annotated my Logstash Pod with:

podAnnotations:
  co.elastic.logs/pipeline: filebeat-7.16.2-logstash-log-pipeline

Hoewever this does not seem to have any effect on the incoming logs. they are still plain text in the message.

Any hints what I am doing wrong?

Thanks

[mtojek (Marcin Tojek)]

It might be easier to guess what's wrong if you share the entire configuration.

[Kosodrom (Kosmodrom)]

Thanks for the response:

logstash.yml

node.name: logstash-dev
    http.host: 0.0.0.0
    log.level: info
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: '${USER}'
    xpack.monitoring.elasticsearch.password: '${PASS}'
    xpack.monitoring.elasticsearch.hosts: ["https://node1:9200", "https://node2:9200", "https://node3:9200"]
    xpack.monitoring.elasticsearch.ssl.certificate_authority: certs/elasticca.crt
    xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
    #Setting pipeline.ecs_compatibility: disabled. Not setting this (either disable or enable with specific version) is depricated
    pipeline.ecs_compatibility: disabled

filebeat.yml

      http.enabled: true
      http.host: localhost
      logging.level: info
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            hints.enabled: true
            add_resource_metadata:
              namespace:
                include_annotations: ["nsannotation1"]
            hints.default_config:
              type: container
              paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log

      filebeat.inputs:
        - type: log
          paths:
            - /var/log/kube-apiserver/*.log
            - /var/log/oauth-apiserver/*.log
          fields:
              log_type: audit
          processors:
            - decode_json_fields:
                fields: ["message"]
                process_array: false
                target: ""
                overwrite_keys: true
                add_error_key: true
				
      processors:
        - add_cloud_metadata:
		
      output.logstash:
        hosts: ['logstash01:5025','logstash2:5025']
        ssl.certificate_authorities: ["certs/filebeat/elasticca.crt"]
        loadbalance: true
        ssl.certificate: "filebeat.crt"
        ssl.key: "filebeat.key"
        ssl.key_passphrase: ${cert_passphrase}
        workers: 2

for running Logstash pipeline I am using this helm chart: helm-charts/logstash at 7.16 · elastic/helm-charts · GitHub

And the ingest pipeline I am trying to use was pushed to Elastic by a Logstash module (from another host)

[Kosodrom (Kosmodrom)]

Could it be that it is not possible to use an ingest pipeline inside annotations:

podAnnotations:
  co.elastic.logs/pipeline: filebeat-7.16.2-logstash-log-pipeline

without also enabling the module?

podAnnotations:
  co.elastic.logs/pipeline: filebeat-7.16.2-logstash-log-pipeline
  co.elastic.logs/module: logstash

[Kosodrom (Kosmodrom)]

I got it working. The precondition is that the ingest pipeline used by the module is existing and (if Logstash is used to ingest to elastic) it considers the pipeline_id in the output.

Now I am able to annotate my workloads with

podAnnotations:
  co.elastic.logs/module: logstash

and it works.

However now I am trying to annotate my workload with a custom pipeline and it seems not to have any effect at all:

podAnnotations:
  co.elastic.logs/pipeline: filebeat-7.16.2-logstash-log-pipeline

Does anyone know why?