Automate applying a pipeline to documents

Elastic Stack Elasticsearch
1

I created "mypipeline" pipeline with grok processor.
I have syslog-ng-(xx) indexes created daily.
If I apply in console:
POST syslog-ng*/_update_by_query?pipeline=mypipeline
I can see all my indexes get updated and showing up on "Discover" page with new "groked" fields.
How can I automate whenever index gets new set of data from syslog-ng pipeline is automatically applied ?

2

Take a look at Ingest pipelines | Elasticsearch Guide [7.13] | Elastic

3

Thanks Mark, that is what I used when got stuck exactly on that spot. Not sure where/how to configure automation step.

Regards

4

It's not clear what you mean by automation step sorry.

5

You can specify a default ingest pipeline through an index template I believe.

6

maybe I am using wrong wording here.
All I need is whenever index gets new set of data from syslog-ng to trigger pipeline with grok pattern and update index.
@Christian_Dahlqvist
I tried with index template but haven't succeeded making it to work (Index gets data without pipeline applied). Only way I was able to get data from pipeline is by doing it manually in console as mentioned. I bet I can use cron to apply changes all the time
POST syslog-ng*/_update_by_query?pipeline=mypipeline
but don't think that is right way.
Regards

7

Which version of Elasticsearch are you using?

You should be able to define a default_pipeline in an index template and have this apply automatically. You can add it by altering index settings for existing indices as well.

8

Thanks Christian,
I am using latest version 7.13.1
Yes that is where I fail. I cannot find example how to do that. I need step by step (as wizard in index_template) to know what and where to put. I created new index_template, put syslog-ng* in Index patterns field and added next in "index settings" step:

{
  "index": {
    "lifecycle": {
      "name": "PanosLog"
    },
    "codec": "best_compression",
    "mapping": {
      "total_fields": {
        "limit": "10000"
      }
    },
    "refresh_interval": "5s",
    "number_of_shards": "1",
    "query": {
      "default_field": [
        "panos.opaque"
      ]
    },
    "default_pipeline": "mypipeline",
    "number_of_routing_shards": "30"
  }
}

but it does not work (indices do not get updated after new data are received).