I have a Logstash pipeline pipeline running in kubernetes. I am collecting all logs from the Kubernetes cluster using filebeat with activated hit based autodiscover. I would like to force filebeat to use a specific ingest pipeline for my Logstash pipeline logs.
In order to achieve this I have annotated my Logstash Pod with:
Hoewever this does not seem to have any effect on the incoming logs. they are still plain text in the message.
Any hints what I am doing wrong?
It might be easier to guess what's wrong if you share the entire configuration.
Thanks for the response:
xpack.monitoring.elasticsearch.hosts: ["https://node1:9200", "https://node2:9200", "https://node3:9200"]
#Setting pipeline.ecs_compatibility: disabled. Not setting this (either disable or enable with specific version) is depricated
- type: kubernetes
- type: log
for running Logstash pipeline I am using this helm chart: helm-charts/logstash at 7.16 · elastic/helm-charts · GitHub
And the ingest pipeline I am trying to use was pushed to Elastic by a Logstash module (from another host)
Could it be that it is not possible to use an ingest pipeline inside annotations:
without also enabling the module?
I got it working. The precondition is that the ingest pipeline used by the module is existing and (if Logstash is used to ingest to elastic) it considers the pipeline_id in the output.
Now I am able to annotate my workloads with
and it works.
However now I am trying to annotate my workload with a custom pipeline and it seems not to have any effect at all:
Does anyone know why?