# Automatic deleting the documents from indices **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2019, 5:16am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097 "2019-05-08T05:16:47Z")\ **Posts on this page:** 10\ **Page:** 2
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 5:59am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/21 "2019-05-09T05:59:27Z")
Now I am using the pipleines.yml to import multiple logs from different locations, so even if i use curator or ILM to delete the index how will I create new index? Will i have to run the pipeline.yml again? or it will dynamically creates the indices???? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 6:03am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/22 "2019-05-09T06:03:24Z")
If you are not using time-based indices you can not use Curator or ILM. You will need to create your own cron job to delete documents using delete-by-query. If you are not updating documents I would however recommend you switch to time-based indices, where the time period covered by each index depends on your data volumes per day and how long you want to keep the data. Specifying `index => "logstash_apchelogs-%{+YYYY.MM}"` in the elasticsearch output will e.g. create a new index every month. You can also create daily indices if you have a shorter retention period by changing the date pattern to `%{+YYYY.MM.dd}`. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 6:04am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/23 "2019-05-09T06:04:53Z")
Wait let me do it practically...will let you know in few minutes ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 6:09am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/24 "2019-05-09T06:09:33Z")
Can you help me with this following error: `[INFO] 2019-05-09 11:50:25.966 [[squidlogs]>worker0] elasticsearch - retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 6:11am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/25 "2019-05-09T06:11:29Z")
That typically means you are running out of disk space and have filled \> 95% of disk, See [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/disk-allocator.html) for how to rectify. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 6:16am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/26 "2019-05-09T06:16:44Z")
What I just tried was I've executed pipelines.yml and a new index was created named "logstash\_apaachelogs" and then I manually deleted it using management tab. Now when I updated few logs in that location then automatically new index was created with above name. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 6:16am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/27 "2019-05-09T06:16:56Z")
Thanks will manage the space ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 6:31am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/28 "2019-05-09T06:31:24Z")
> [@Vikash\_Singh1](#): > > What I just tried was I've executed pipelines.yml and a new index was created named "logstash\_apaachelogs" and then I manually deleted it using management tab. Now when I updated few logs in that location then automatically new index was created with above name. What I just tried was I've executed pipelines.yml and a new index was created named "logstash\_apaachelogs" and then I manually deleted it using management tab. Now when I updated few logs in that location then automatically new index was created with above name. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 10, 2019, 4:40am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/29 "2019-05-10T04:40:44Z")
While using the ILM I am getting this error: { "indices" : { "logstash\_apchelogs-2019.05" : { "index" : "logstash\_apchelogs-2019.05", "managed" : true, "policy" : "IndexManagement", "lifecycle\_date\_millis" : 1557400555402, "phase" : "hot", "phase\_time\_millis" : 1557400854567, "action" : "rollover", "action\_time\_millis" : 1557400854567, "step" : "ERROR", "step\_time\_millis" : 1557401400805, "failed\_step" : "check-rollover-ready", "step\_info" : { "type" : "illegal\_argument\_exception", "reason" : "index name [logstash\_apchelogs-2019.05] does not match pattern '^.\*-\d+'", "stack\_trace" : """ java.lang.IllegalArgumentException: index name [logstash\_apchelogs-2019.05] does not match pattern '^.\*-\d+' at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.generateRolloverIndexName(TransportRolloverAction.java:237) at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:120) at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:67) at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:124) at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:211) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:140) at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:208) at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:164) at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:141) at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:97) at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:161) at org.elasticsearch.xpack.indexlifecycle.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:55) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1269) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1777) at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:115) at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:128) at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:244) at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:199) at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:164) at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:192) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) """ }, "phase\_execution" : { "policy" : "IndexManagement", "phase\_definition" : { "min\_age" : "0ms", "actions" : { "rollover" : { "max\_size" : "50gb", "max\_age" : "30d" } } }, "version" : 2, "modified\_date\_in\_millis" : 1557393705856 } } } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 7, 2019, 4:40am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/30 "2019-06-07T04:40:44Z")
Closed on June 7, 2019, 4:40am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097.md?page=1)