Custom Ingest Pipeline

[acosta353]

Hello,

I'm new to Elastic, and I'm trying to search how to configure properly a custom ingest pipeline to Apache integration.

In fact, I need to add some different groks and time formats so that some logs can be parsed properly, and I can see that an ingest pipeline named "logs-apache.access@custom" is called in the end of "logs-apache.access-1.7.0" pipeline.

I am not understanding how can I add these new groks and timeformats, in order to parse all logs that I need using this custom ingest pipeline so that it won't be overwritten by any updates on main apache integration pipelines.

Can someone help me with this?

Thanks!

[Barry_Solomon]

I guess one thing you can do is open that pipeline and save it as something else, different name, so you are cloning it. The modify it - remove any processor you don't need and adjust any in there to suit your needs. Save it and then make sure you have an index template (see documentation) setup that your new logs indices can use (set the pattern, such as logs*, put in any setting, mappings and aliases you might want to use. The back in piipeline, find yours and find the area to attach it to an index template. You are set. Only one template can be applied to any matching index but you could have multiple pipelines apply. Test it to make sure it does what you expect. Always consult Elastic documentation when in doubt - it is superb

[acosta353]

Hello!

Thanks for you answer.

In the end, I managed to create a new index template and ingest pipeline for the grok that I want, and now I'm able to parse the logs that I wanted to.

Regards