# Are ingest pipelines created by agent expected to be different from the same type of pipeline created by beats?

**URL:** <https://discuss.elastic.co/t/are-ingest-pipelines-created-by-agent-expected-to-be-different-from-the-same-type-of-pipeline-created-by-beats/345313>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 18, 2023, 4:33pm UTC](https://discuss.elastic.co/t/are-ingest-pipelines-created-by-agent-expected-to-be-different-from-the-same-type-of-pipeline-created-by-beats/345313 "2023-10-18T16:33:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [October 18, 2023, 4:33pm UTC](https://discuss.elastic.co/t/are-ingest-pipelines-created-by-agent-expected-to-be-different-from-the-same-type-of-pipeline-created-by-beats/345313/1 "2023-10-18T16:33:12Z")

</div>

So, I've been assuming that the ingest pipelines created by Elastic Agent and those created by Filebeat, would be pretty much identical. I expected some differences, but nothing that would make them output different fields.

Today I found that there are at least some major differences in the Apache ingest pipelines. Specifically, the one from Elastic Agent has more grok patterns than the one from Filebeat.

The end result is that Filebeat doesn't support the same log formats that the Agent integration supports.

Filebeat's grok processor:

```json
{
    "grok": {
      "field": "event.original",
      "patterns": [
        "%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:_tmp.url_orig} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?",
        "%{IPORHOST:source.address} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:_tmp.url_orig} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?",
        "%{IPORHOST:source.address} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"-\" %{NUMBER:http.response.status_code:long} -",
        "\\[%{HTTPDATE:apache.access.time}\\] %{IPORHOST:source.address} %{DATA:apache.access.ssl.protocol} %{DATA:apache.access.ssl.cipher} \"%{WORD:http.request.method} %{DATA:_tmp.url_orig} HTTP/%{NUMBER:http.version}\" (-|%{NUMBER:http.response.body.bytes:long})"
      ],
      "ignore_missing": true
    }
  }

```

Agents grok processor:

```json
{
    "grok": {
      "field": "event.original",
      "patterns": [
        "(%{IPORHOST:destination.domain} )?%{IPORHOST:source.address} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:_tmp.url_orig} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?( X-Forwarded-For=\"%{ADDRESS_LIST:apache.access.remote_addresses}\")?",
        "%{IPORHOST:source.address} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"-\" %{NUMBER:http.response.status_code:long} -",
        "\\[%{HTTPDATE:apache.access.time}\\] %{IPORHOST:source.address} %{DATA:apache.access.ssl.protocol} %{DATA:apache.access.ssl.cipher} \"%{WORD:http.request.method} %{DATA:_tmp.url_orig} HTTP/%{NUMBER:http.version}\" (-|%{NUMBER:http.response.body.bytes:long})"
      ],
      "ignore_missing": true,
      "pattern_definitions": {
        "ADDRESS_LIST": "(%{IP})(\"?,?\\s*(%{IP}))*"
      }
    }
  }

```

So far I've just investigated the apache access pipelines. But I'm guessing there could be differences for other types of pipelines.

Is this expected? If so, why?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 18, 2023, 4:39pm UTC](https://discuss.elastic.co/t/are-ingest-pipelines-created-by-agent-expected-to-be-different-from-the-same-type-of-pipeline-created-by-beats/345313/2 "2023-10-18T16:39:42Z")

</div>

> [@jerrac](#):
>
> Is this expected? If so, why?

Assuming that the Elastic plan is to replace the beats family with Elastic Agent and the Filebeat modules with Elastic Agent Integrations, I don't think that the filebeat modules are getting the same attention as the Elastic Agent Integrations and some changes on the ingest pipelines of the Integrations are not reflected on the Filebeat Modules.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [October 18, 2023, 4:43pm UTC](https://discuss.elastic.co/t/are-ingest-pipelines-created-by-agent-expected-to-be-different-from-the-same-type-of-pipeline-created-by-beats/345313/3 "2023-10-18T16:43:49Z")

</div>

I'd believe that, but the fact that Agent doesn't support hints based auto-discovery while Filebeat does kinda makes me wonder.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2023, 6:44pm UTC](https://discuss.elastic.co/t/are-ingest-pipelines-created-by-agent-expected-to-be-different-from-the-same-type-of-pipeline-created-by-beats/345313/4 "2023-11-15T18:44:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.