Hello,
We are trying to find a way to manage custom ingest pipelines and fields/mappings for our integrations.
As an example, we are using the Cisco Duo integration, this integration has the following ingest pipelines.
logs-cisco_duo.admin-1.5.1
logs-cisco_duo.auth-1.5.1
logs-cisco_duo.offline_enrollment-1.5.1
logs-cisco_duo.summary-1.5.1
logs-cisco_duo.telephony-1.5.1
Each one of these ingest pipelines would call a custom pipeline named logs_cisco_duo.DATASET@custom, if we want to add or remove some field we would need to create, edit and manage at least five pipelines.
Is there an easy way to do that to all the pipelines on a integration?
Another issue we had is that we created a custom ingest pipeline for the admin logs, called logs-cisco_duo.admin@custom, because we needed to extract the value of a field and create another one.
We created the following processor:
{
"set": {
"field": "source.ip",
"copy_from": "cisco_duo.admin.flattened.ip_address",
"ignore_empty_value": true,
"if": "ctx?.event?.action == \"admin_2fa_error\"",
"ignore_failure": true
}
}
This worked as expected, but for our surprise we now have an alert of conflicting fields for an ecs field.
ip - .ds-logs-cisco_duo.auth-duo-2022.10.19-000001, .ds-logs-cisco_duo.auth-duo-2022.11.18-000002
keyword - .ds-logs-cisco_duo.admin-duo-2022.11.18-000002
It says that in some indices it is mapped as an ip field and in the new index where we added the field it got mapped as a keyword field.
Shouldn't the integrations have the mappings for the ECS fields per default?
This means that if We want to add a custom field, even if it is an ECS field, we would need to edit and manage five custom ingest pipelines and edit and manage five templates.
With more integrations, this number would escalate pretty quickly.
Paired with this another issue we had, I'm failing to see what is the advantage of Elastic Agent integrations, it seems more and more that if you want to customize anything you should avoid Elastic Agent and build your owns pipeplines to parse.