# Custom Logs Ingest Pipeline **URL:** \ **Category:** Elasticsearch\ **Tags:** fleet\ **Created:** [January 14, 2022, 12:06pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409 "2022-01-14T12:06:00Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jsteenkamp](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jsteenkamp](https://discuss.elastic.co/u/jsteenkamp)\ **Post date:** [January 14, 2022, 12:06pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/1 "2022-01-14T12:06:00Z")
@ruflin This is more or less a duplicate of [Fleet Custom Logs Ingest Pipeline](https://discuss.elastic.co/t/fleet-custom-logs-ingest-pipeline/262999), but I still find it hard to figure out which steps I have to follow. I would really appreciate better documentation or some useful directions. I am also trying to ingest a log file using the custom log integration in Fleet. Is there a way to configure a custom ingest pipeline as well so I can parse the custom fields in the message field using grok patterns? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 14, 2022, 1:06pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/2 "2022-01-14T13:06:59Z")
@joshdover You have been recently working on some ideas to improve this further. Do you have by chance some links that can be shared? @mostlyjason For awareness of the discussion. ---
**Author:** ![joshdover](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshdover/32/42020_2.png) [@joshdover](https://discuss.elastic.co/u/joshdover)\ **Post date:** [January 17, 2022, 2:04pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/3 "2022-01-17T14:04:30Z")
We are planning some changes to how we setup the templates to make user customizations much easier to add (see [kibana#121118](https://github.com/elastic/kibana/issues/121118)), however this won't fully solve the custom ingest pipeline case. The best option I have right now is to edit the existing `logs-log.log@custom` component and then rollover the data stream: 1. Create a new ingest pipeline 2. Edit the `logs-log.log@custom` component template to add the `default_pipeline` index setting to point to the newly created ingest pipeline 3. Rollover any existing data streams that match `logs-log.log-*` to apply the new settings using the [Rollover API](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/indices-rollover-index.html) ---
**Author:** ![Doommius](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doommius/32/100422_2.png) [@Doommius](https://discuss.elastic.co/u/Doommius)\ **Post date:** [January 17, 2022, 2:57pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/4 "2022-01-17T14:57:42Z")
Hey Josh, I agree with @jsteenkamp . Some better docs on this would be nice. Eg:[Custom logs | Elastic Docs](https://docs.elastic.co/en/integrations/log) how to use ingest pipeline either via the "Custom configurations" in the integration settings, and if its possible to just use the same configuration as in filebeat. eg can I just throw in output.Elasticsearch.pipeline: 'somepipeline' It would be better just adding a optional field in the custom logs integration settings where it's possible to lookup existing pipelines or telling users a ingest pipeline is required etc. ---
**Author:** ![jsteenkamp](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jsteenkamp](https://discuss.elastic.co/u/jsteenkamp)\ **Post date:** [January 17, 2022, 4:15pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/5 "2022-01-17T16:15:18Z")
@joshdover thank you for your suggestions. Based on your input, I have found this [instruction](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html#pipeline-custom-logs-configuration) online and guess what, it works in just two simple steps. 1. create your custom pipeline 2. define your custom pipeline in the custom configurations ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a43e9734ce9b1aaae8eb378bc8549f3d99ed2d1.jpeg) ---
**Author:** ![Doommius](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doommius/32/100422_2.png) [@Doommius](https://discuss.elastic.co/u/Doommius)\ **Post date:** [January 19, 2022, 8:40am UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/6 "2022-01-19T08:40:58Z")
Ty. I’ve been looking for this for a few days and tested a few different things.😂 This should really be added to the custom logs docs as well. @ruflin and @joshdover it is possible to setup a merge request on git or some way to update the docs from the user side ? Looking forward to hearing back. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 20, 2022, 12:44pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/7 "2022-01-20T12:44:34Z")
It could be contributed here: [integrations/packages/log at master · elastic/integrations · GitHub](https://github.com/elastic/integrations/tree/master/packages/log) What you did works, but using the pipeline setting on the input itself is something we don't encourage. The feature is there because we inherit it from beats directly. Instead the pipeline should be set in the settings on the data stream. But as discussed above, we don't provide everything needed here yet. You know pretty well how the stack works and I'm sure it would be simple for you to change from a setting in the yaml to a setting on the data stream as soon as we have it. What I'm worried around documenting it is that many users will start using it and then eventually will be stuck. Maybe we can mention exactly this issue in the docs? ---
**Author:** ![Doommius](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doommius/32/100422_2.png) [@Doommius](https://discuss.elastic.co/u/Doommius)\ **Post date:** [January 20, 2022, 4:57pm UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/8 "2022-01-20T16:57:15Z")
Hey Ruflin, I've setup it up via the components now. if this is the best practice for it at the moment. got this weird issue though, I assume it's with the mappings not being setup correctly? ![Untitled](https://us1.discourse-cdn.com/elastic/original/3X/4/2/42388eee00c305be32cca19507dbd44740d18b21.png) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 24, 2022, 9:34am UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/9 "2022-01-24T09:34:47Z")
Something looks off with the mappings. You must set keyword as the default. Have a look at some of the existing templates installed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 21, 2022, 9:35am UTC](https://discuss.elastic.co/t/custom-logs-ingest-pipeline/294409/10 "2022-02-21T09:35:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.