# Is there an easy way to manage custom ingest pipeline and custom fields for Elastic Agent Integrations?

**URL:** <https://discuss.elastic.co/t/is-there-an-easy-way-to-manage-custom-ingest-pipeline-and-custom-fields-for-elastic-agent-integrations/319682>\
**Category:** Elastic Agent\
**Created:** [November 23, 2022, 7:57pm UTC](https://discuss.elastic.co/t/is-there-an-easy-way-to-manage-custom-ingest-pipeline-and-custom-fields-for-elastic-agent-integrations/319682 "2022-11-23T19:57:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 23, 2022, 7:57pm UTC](https://discuss.elastic.co/t/is-there-an-easy-way-to-manage-custom-ingest-pipeline-and-custom-fields-for-elastic-agent-integrations/319682/1 "2022-11-23T19:57:30Z")

</div>

Hello,

We are trying to find a way to manage custom ingest pipelines and fields/mappings for our integrations.

As an example, we are using the Cisco Duo integration, this integration has the following ingest pipelines.

```auto
logs-cisco_duo.admin-1.5.1 
logs-cisco_duo.auth-1.5.1 
logs-cisco_duo.offline_enrollment-1.5.1 
logs-cisco_duo.summary-1.5.1 
logs-cisco_duo.telephony-1.5.1 

```

Each one of these ingest pipelines would call a custom pipeline named `logs_cisco_duo.DATASET@custom`, if we want to add or remove some field we would need to create, edit and manage at least **five** pipelines.

Is there an easy way to do that to all the pipelines on a integration?

Another issue we had is that we created a custom ingest pipeline for the admin logs, called `logs-cisco_duo.admin@custom`, because we needed to extract the value of a field and create another one.

We created the following processor:

```auto
  {
    "set": {
      "field": "source.ip",
      "copy_from": "cisco_duo.admin.flattened.ip_address",
      "ignore_empty_value": true,
      "if": "ctx?.event?.action == \"admin_2fa_error\"",
      "ignore_failure": true
    }
  }

```

This worked as expected, but for our surprise we now have an alert of conflicting fields for an ecs field.

```auto
ip - .ds-logs-cisco_duo.auth-duo-2022.10.19-000001, .ds-logs-cisco_duo.auth-duo-2022.11.18-000002
keyword - .ds-logs-cisco_duo.admin-duo-2022.11.18-000002

```

It says that in some indices it is mapped as an `ip` field and in the new index where we added the field it got mapped as a `keyword` field.

Shouldn't the integrations have the mappings for the ECS fields per default?

This means that if We want to add a custom field, even if it is an ECS field, we would need to edit and manage **five** custom ingest pipelines and edit and manage **five** templates.

With more integrations, this number would escalate pretty quickly.

Paired with this [another issue](https://discuss.elastic.co/t/is-there-an-easy-way-to-change-the-lifecycle-policy-of-a-fleet-managed-data-stream/316941) we had, I'm failing to see what is the advantage of Elastic Agent integrations, it seems more and more that if you want to customize anything you should avoid Elastic Agent and build your owns pipeplines to parse.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 8, 2022, 7:01pm UTC](https://discuss.elastic.co/t/is-there-an-easy-way-to-manage-custom-ingest-pipeline-and-custom-fields-for-elastic-agent-integrations/319682/2 "2022-12-08T19:01:37Z")

</div>

By suggestion of someone from Elastic I opened this [issue](https://github.com/elastic/kibana/issues/146792) for discussion about this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 5, 2023, 7:01pm UTC](https://discuss.elastic.co/t/is-there-an-easy-way-to-manage-custom-ingest-pipeline-and-custom-fields-for-elastic-agent-integrations/319682/3 "2023-01-05T19:01:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.