Customize ingest pipelines in Fleet / Elastic Agent

[icious]

Hello,

I was fiddling around with Elastic Agent and Fleet to integrate multiple log sources in the same agent, and I am getting "Provided Grok expressions do not match field value" error with Fortinet integration.

Using Filebeat I was able to "bypass" this error modifying Grok pattern to match my logs' format, but when I try to adapt Elastic Agent ingest pipelines the same way, it does not work. I tried modifying them manually from Kibana and Elastic Agent's data folder, but none of them worked.

So my question is, is there a way to modify Grok patterns used in Fleet integrations or should I stick to Filebeat only?

Thank you in advance

[ruflin]

Hi @icious Welcome to the forum.

I have opened a PR here some time ago with some docs related to this: https://github.com/elastic/beats/pull/23247 You can modify the ingest pipeline directly in Elasticsearch and it should work. But be aware that the next update of the package will overwrite it and is not what you want. What you can do as an alternative is describe in the PR.

The modifications you are making, are they specific to your use case or more generic? If more generic, perhaps these could be contributed here to the fortinet package/integration: https://github.com/elastic/integrations/tree/master/packages/fortinet Like this they would be available to everyone in the next version.

[icious]

Hi @ruflin, modifying the ingest pipeline directly from Elasticsearch has worked fine. Thank you!