Ingestion pipelines for new indices created everyday

[katakam_chaitanya]

I have new indices created on my Elasticsearch cluster everyday.
I want to take all my docs through a ingestion pipeline.
But as new indices get created automatically as date changes ( index name example:cwl-12.12.2022), how can I write set a default ingestion pipeline to all new indices getting created as well ??

Thank you

[ibra_013 (Ibra13)]

Hi,

Can you describe your architecture? and what version you stack

filebeat-->Elasticsearch for example .

[katakam_chaitanya]

Hi ibra,
Our application is deployed in an ecs cluster and we are firstly shipping logs from our aws to cloudwatch logs and then through subscription filters we are sending these logs to Elasticsearch which is also in aws(opensearch service).

Now, everyday a new index gets created and the problem I have talked above is regarding this.

[system]

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[warkolm (Mark Walkom)]

If cloudwatch is sending these to your opensearch you will need to ask aws, as none of those are our products.