Automate applying a pipeline to documents

I created "mypipeline" pipeline with grok processor.
I have syslog-ng-(xx) indexes created daily.
If I apply in console:
POST syslog-ng*/_update_by_query?pipeline=mypipeline
I can see all my indexes get updated and showing up on "Discover" page with new "groked" fields.
How can I automate whenever index gets new set of data from syslog-ng pipeline is automatically applied ?

Take a look at Ingest pipelines | Elasticsearch Guide [7.13] | Elastic

Thanks Mark, that is what I used when got stuck exactly on that spot. Not sure where/how to configure automation step.


It's not clear what you mean by automation step sorry.

You can specify a default ingest pipeline through an index template I believe.

maybe I am using wrong wording here.
All I need is whenever index gets new set of data from syslog-ng to trigger pipeline with grok pattern and update index.
I tried with index template but haven't succeeded making it to work (Index gets data without pipeline applied). Only way I was able to get data from pipeline is by doing it manually in console as mentioned. I bet I can use cron to apply changes all the time
POST syslog-ng*/_update_by_query?pipeline=mypipeline
but don't think that is right way.

Which version of Elasticsearch are you using?

You should be able to define a default_pipeline in an index template and have this apply automatically. You can add it by altering index settings for existing indices as well.

Thanks Christian,
I am using latest version 7.13.1
Yes that is where I fail. I cannot find example how to do that. I need step by step (as wizard in index_template) to know what and where to put. I created new index_template, put syslog-ng* in Index patterns field and added next in "index settings" step:

  "index": {
    "lifecycle": {
      "name": "PanosLog"
    "codec": "best_compression",
    "mapping": {
      "total_fields": {
        "limit": "10000"
    "refresh_interval": "5s",
    "number_of_shards": "1",
    "query": {
      "default_field": [
    "default_pipeline": "mypipeline",
    "number_of_routing_shards": "30"

but it does not work (indices do not get updated after new data are received).

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.