Filebeat module ingest pipeline not working in logstash

Hi Team,
I am using apache module and fileset in Beats+ELK stack where Filebeat is sending logs to logstash, logstash is using Ingest pipeline(we get module ingest pipeline - filebeat-8.7.1-apache-access-pipeline) , but it is somehow not utilizing the pipeline when ingesting data into elasticsearch. The same ingest pipeline works perfectly when I use filebeat directly to elasticsearch but not when logstash is used in between.

Expectation: Data plugin/geoip/useragent plugins should be applied to the data which is part of ingest pipeline. The _simulate api gives the expeted output but not via logstash.

Whole ELK + Beats is on version 8.7.1c

Expected output - 30th may 2014 - When pushing data directly from filebeat (Uses elasticsearch as output in filebeat.yml)

Filebeat.yml -

Logstash logs: Data is coming to logstash, and I can see pipeline name is also mentioned, but it is not being utilized in Elasticsearch- No geoip or user agent filter got applied

Kibana output: Faulty output - 17th july data is coming from filebeat-->logstash->elasticsearch
Exppected output - 30th may 2014 - data is pushed directly through filebeat(with output set to elasticsearch and pipeline name mentioned in filebeat.yml)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.