# Apache Logs Not Parsing with Filebeat Ingestion Pipeline

**URL:** <https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 28, 2023, 9:09pm UTC](https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968 "2023-10-28T21:09:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![huzaifa224](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huzaifa224/32/105868_2.png) [@huzaifa224](https://discuss.elastic.co/u/huzaifa224)\
**Post date:** [October 28, 2023, 9:09pm UTC](https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968/1 "2023-10-28T21:09:38Z")

</div>

I have installed Filebeat version 8.10.2 on an Ubuntu server and configured it to send Apache access and error logs to Logstash. While the logs are displayed in Kibana, they are not parsing through their default ingest pipelines, causing issues with data visualization on the dashboard.

To set up Filebeat, I followed the instructions provided in the official documentation, along with additional articles for more specific guidance:

1. [Filebeat Installation and Configuration](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html)

I also followed these three articles to load the necessary components for proper log processing:

1. [Load Index Template Manually](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually)
2. [Load Kibana Dashboards](https://www.elastic.co/guide/en/beats/filebeat/current/load-kibana-dashboards.html)
3. [Load Ingest Pipelines](https://www.elastic.co/guide/en/beats/filebeat/current/load-ingest-pipelines.html)

Additionally, I referred to [this Logstash documentation](https://www.elastic.co/guide/en/logstash/8.10/use-ingest-pipelines.html).

Here are the commands I executed for better clarity:

LOAD INDEX TEMPLATE:

filebeat setup --index-management -E output.logstash.enabled=false -E output.elasticsearch.username="elastic" -E output.elasticsearch.password="KkXXNnUzgahp" -E 'output.elasticsearch.hosts=["[http://X.X.X.X:9200](http://X.X.X.X:9200)"]

LOAD KIBANA DASHBOARDS:

filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.username="elastic" -E output.elasticsearch.password="zgXahkXNnUp" -E output.elasticsearch.hosts=["[http://X.X.X.X:9200](http://X.X.X.X:9200)"] -E setup.kibana.host=[http://X.X.X.X:5601](http://X.X.X.X:5601)

LOAD PIPELINE:

filebeat setup --pipelines --modules apache --force-enable-module-filesets

I assumed that after loading the default ingest pipelines, the Apache error and access logs would be parsed without the need for additional configuration in Logstash. However, the logs are still not parsing as expected after applying the default ingestion pipeline.

 ![ELK-APACHE-LOGS](https://us1.discourse-cdn.com/elastic/original/3X/3/5/3593f73da2f832df9f55113a09c081ac27a384ad.png)

Here is the detail of log:

 ![log-p1](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75f4e06ad0b7bfadcea783c3e095f8798d6df1be.png)  
 ![log-p2](https://us1.discourse-cdn.com/elastic/original/3X/8/2/8255d6045b2bdd4bcc7df541140b7d180d1002a4.png)

Here is logstash config file:

> input {  
> beats {  
> port =\> 5044  
> }  
> }  
> output {  
> if [@metadata][pipeline] {  
> elasticsearch {  
> hosts =\> "[http://x.x.x.x:9200](http://x.x.x.x:9200)"  
> manage\_template =\> false  
> index =\> "%{[@metadata][beat]}-%{[@metadata][version]}"  
> action =\> "create"  
> pipeline =\> "%{[@metadata][pipeline]}"  
> user =\> "elastic"  
> password =\> "XXzgahpNnU"  
> }  
> } else {  
> elasticsearch {  
> hosts =\> "[http://x.x.x.x:9200](http://x.x.x.x:9200)"  
> manage\_template =\> false  
> index =\> "%{[@metadata][beat]}-%{[@metadata][version]}"  
> action =\> "create"  
> user =\> "elastic"  
> password =\> "XXzgahpNnU"  
> }  
> }  
> }

Despite following the steps and configurations mentioned, the logs are not being parsed correctly, as shown in the provided images. Required experts guidance.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 29, 2023, 3:55pm UTC](https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968/3 "2023-10-29T15:55:02Z")

</div>

Hi @huzaifa224

My recommendation is to always get the integration working directly from

Filebeat -\> Elasticsearch

Directly first and then try to put logstash in the middle.

Also, instead of running all those separate setup commands, it's much easier and less error prone just to run

`filebeat setup -e`

Which loads all the assets in one command.

So I would clean up all the indices and run filebeat directly to elastic first and see if that works.

Once that works then try logstash in the middle.

Another issue could be is if you're Apache logs are a non-standard format.

---

<div class="post-metadata">

**Author:** ![huzaifa224](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huzaifa224/32/105868_2.png) [@huzaifa224](https://discuss.elastic.co/u/huzaifa224)\
**Post date:** [October 30, 2023, 11:36am UTC](https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968/4 "2023-10-30T11:36:41Z")

</div>

i also tried to send data directly to elasticsearch which was working fine, but i need to use logstash in between but it give issues while using it. Does the command "filebeat setup -e" will load index template, dashboards, pipelines?  
the apache logs are in standard format

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 30, 2023, 12:37pm UTC](https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968/5 "2023-10-30T12:37:39Z")

</div>

I think that the ingest pipeline is breaking right [here](https://github.com/elastic/beats/blob/adcd4b0c7cc2ef43b4b95ff5ed73d2c9f5a58548/filebeat/module/apache/access/ingest/pipeline.yml#L7C1-L9C33).

A lot of Elasticsearch Ingest Pipelines will try to rename the `message` field to `event.original`, and since they do not expect the source message to already have an `event.original` field, this processor will fail and the ingest pipeline will not work.

From version 8.X Logstash comes with ecs compatibility enabled per default, and this will make some inputs to create the field `event.original`, which will then break the ingest pipelines.

This is a [know issue](https://github.com/elastic/integrations/issues/3451).

You can try the following:

- set `pipeline.ecs_compatibility` to `disabled` on `logstash.yml` or for the specific pipeline in the `pipelines.yml`
- add the setting `enrich => none` in your beats output.

This will disable the ecs compatibility and tell logstash to not add any fields to the source event, this way the ingest pipeline should work as expected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 27, 2023, 2:38pm UTC](https://discuss.elastic.co/t/apache-logs-not-parsing-with-filebeat-ingestion-pipeline/345968/6 "2023-11-27T14:38:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.