Filebeat module ingest pipeline not working in logstash

[nbindal (Nishant)]

Hi Team,
I am using apache module and fileset in Beats+ELK stack where Filebeat is sending logs to logstash, logstash is using Ingest pipeline(we get module ingest pipeline - filebeat-8.7.1-apache-access-pipeline) , but it is somehow not utilizing the pipeline when ingesting data into elasticsearch. The same ingest pipeline works perfectly when I use filebeat directly to elasticsearch but not when logstash is used in between.

Expectation: Data plugin/geoip/useragent plugins should be applied to the data which is part of ingest pipeline. The _simulate api gives the expeted output but not via logstash.

Versions:
Whole ELK + Beats is on version 8.7.1c

[nbindal (Nishant)]

Expected output - 30th may 2014 - When pushing data directly from filebeat (Uses elasticsearch as output in filebeat.yml)

[nbindal (Nishant)]

Filebeat.yml -

[nbindal (Nishant)]

Logstash logs: Data is coming to logstash, and I can see pipeline name is also mentioned, but it is not being utilized in Elasticsearch- No geoip or user agent filter got applied

[nbindal (Nishant)]

Kibana output: Faulty output - 17th july data is coming from filebeat-->logstash->elasticsearch
Exppected output - 30th may 2014 - data is pushed directly through filebeat(with output set to elasticsearch and pipeline name mentioned in filebeat.yml)