Filebeat is not using custom pipeline mentioned in the output.elasticsearch

[Mahesh_Gadagi]

Hello all,

Currently, I am using filebeat, elastic search 7.9.1 version and I have added the custom Ingest pipeline to push the logs but still, filebeat creates a new Ingest pipeline as filebeat-7.9.1-haproxy-log-pipeline. But whenever filebeat pushes the logs I want it to push the pipeline I have mentioned in the configuration, can someone help, please?

filebeat.yml

output.elasticsearch:

  hosts: ['localhost:9200']

  username: 'abc'

  password: 'abc'

  index: "mytestindex-%{[agent.version]}-%{+yyyy.MM.dd}"

  pipeline: "my_test_pipeline"

Here I am expecting it should use my_test_pipeline to push the logs instead of filebeat-7.9.1-haproxy-log-pipeline

Regards,
Mahesh

[rugenl (Len Rugen)]

Get node stats

GET _nodes/your-ingest-node/stats/ingest

Check the count fields in "my_test_pipeline"

[Mahesh_Gadagi]

How this will help me to solve the problem, please ?

[rugenl (Len Rugen)]

If the count is > 0 entering the pipeline, the data is getting there. Each process in the pipeline has a count, so you can see some of what is happening inside the pipeline.

[Mahesh_Gadagi]

But I already know that none of the logs are processing through my pipeline.