# Automate applying a pipeline to documents **URL:** \ **Category:** Elasticsearch\ **Tags:** ingest-pipeline\ **Created:** [June 15, 2021, 10:04pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059 "2021-06-15T22:04:09Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Spricer](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@Spricer](https://discuss.elastic.co/u/Spricer)\ **Post date:** [June 15, 2021, 10:04pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/1 "2021-06-15T22:04:09Z")
I created "mypipeline" pipeline with grok processor. I have syslog-ng-(xx) indexes created daily. If I apply in console: `POST syslog-ng*/_update_by_query?pipeline=mypipeline` I can see all my indexes get updated and showing up on "Discover" page with new "groked" fields. How can I automate whenever index gets new set of data from syslog-ng pipeline is automatically applied ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 15, 2021, 11:20pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/2 "2021-06-15T23:20:29Z")
Take a look at [Ingest pipelines | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/ingest.html#set-default-pipeline) ---
**Author:** ![Spricer](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@Spricer](https://discuss.elastic.co/u/Spricer)\ **Post date:** [June 16, 2021, 4:13am UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/3 "2021-06-16T04:13:54Z")
Thanks Mark, that is what I used when got stuck exactly on that spot. Not sure where/how to configure automation step. Regards ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2021, 4:26am UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/4 "2021-06-16T04:26:46Z")
It's not clear what you mean by automation step sorry. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2021, 5:31am UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/5 "2021-06-16T05:31:17Z")
You can specify a default ingest pipeline through an index template I believe. ---
**Author:** ![Spricer](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@Spricer](https://discuss.elastic.co/u/Spricer)\ **Post date:** [June 16, 2021, 8:02pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/6 "2021-06-16T20:02:26Z")
maybe I am using wrong wording here. All I need is whenever index gets new set of data from syslog-ng to trigger pipeline with grok pattern and update index. @Christian_Dahlqvist I tried with index template but haven't succeeded making it to work (Index gets data without pipeline applied). Only way I was able to get data from pipeline is by doing it manually in console as mentioned. I bet I can use cron to apply changes all the time `POST syslog-ng*/_update_by_query?pipeline=mypipeline` but don't think that is right way. Regards ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2021, 8:23pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/7 "2021-06-16T20:23:07Z")
Which version of Elasticsearch are you using? You should be able to define a [default\_pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/index-modules.html) in an index template and have this apply automatically. You can add it by altering index settings for existing indices as well. ---
**Author:** ![Spricer](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@Spricer](https://discuss.elastic.co/u/Spricer)\ **Post date:** [June 17, 2021, 2:41pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/8 "2021-06-17T14:41:25Z")
Thanks Christian, I am using latest version 7.13.1 Yes that is where I fail. I cannot find example how to do that. I need step by step (as wizard in index\_template) to know what and where to put. I created new index\_template, put `syslog-ng*` in `Index patterns` field and added next in "index settings" step: ```auto { "index": { "lifecycle": { "name": "PanosLog" }, "codec": "best_compression", "mapping": { "total_fields": { "limit": "10000" } }, "refresh_interval": "5s", "number_of_shards": "1", "query": { "default_field": [ "panos.opaque" ] }, "default_pipeline": "mypipeline", "number_of_routing_shards": "30" } } ``` but it does not work (indices do not get updated after new data are received). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2021, 2:42pm UTC](https://discuss.elastic.co/t/automate-applying-a-pipeline-to-documents/276059/9 "2021-07-15T14:42:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.