Cef log with custom udp integration

[Cristina_Marletta_Li (Cristina Marletta Livi)]

Hi all,
I am using Custom UDP integration to do CEF log ingestion. I cannot use CEF integration due to limitations of that integration. The decode_cef processor is not available in the pipelines. What is the best way to do CEF parsing with custom UDP integration without writing a costly and detailed custom pipeline?

[leandrojmp (Leandro Pereira)]

@Cristina_Marletta_Li

The decode_cef processor is not available in the pipelines.

The decode_cef is a Filebeat processor, not an Ingest processor, to use it in the Custom UDP integration you need to add it to the Processor lists in the Advanced options part.

Something like this:

@Cristina_Marletta_Li

I cannot use CEF integration due to limitations of that integration.

What limitations? The Custom CEF integration is basically a custom tcp, custom udp or custom log integration with some built-in processors, like a rename to rename message into event.original and a decode_cef processor.

The udp input for the Custom CEF integration is defined here.

[Cristina_Marletta_Li (Cristina Marletta Livi)]

In fact, you cannot rename the dataset name in CEF integration.
If you rename the dataset, you cannot associate a custom ingestion pipeline.

[leandrojmp (Leandro Pereira)]

@Cristina_Marletta_Li

In fact, you cannot rename the dataset name in CEF integration.

I didn't tested, but you can change it here:

@Cristina_Marletta_Li

If you rename the dataset, you cannot associate a custom ingestion pipeline.

Can you share the agente configuration after you created it? You normally have the option to add a custom ingest pipeline in all integrations, but sometimes this only shows up after the integration is created.

Also, if you use a custom dataset name like logs-custom.dataset-namespace, you need to have a custom template where you can set a custom ingest pipeline.

[Cristina_Marletta_Li (Cristina Marletta Livi)]

I opened a case to support about it. The problem seems related to this:

[Fleet] Packages with custom dataset use the fallback index templates

Basically, whenever we change the dataset name to something that default index template (logs-cef.log), index pattern logs-cef.log-* doesn't cover, this dataset/indices will never get the expected ingestion pipelines.
I tried to hange the dataset name to something that can match the index pattern, for example: logs-cef.log-prod. By doing this, the agent stopped ingesting events. Only by renaming the dataset name back to cef.log did events start coming in again. I had no problems with Custom UDP integration.