Now I am using the pipleines.yml to import multiple logs from different locations, so even if i use curator or ILM to delete the index how will I create new index? Will i have to run the pipeline.yml again? or it will dynamically creates the indices????
If you are not using time-based indices you can not use Curator or ILM. You will need to create your own cron job to delete documents using delete-by-query.
If you are not updating documents I would however recommend you switch to time-based indices, where the time period covered by each index depends on your data volumes per day and how long you want to keep the data. Specifying index => "logstash_apchelogs-%{+YYYY.MM}" in the elasticsearch output will e.g. create a new index every month. You can also create daily indices if you have a shorter retention period by changing the date pattern to %{+YYYY.MM.dd}.
Wait let me do it practically...will let you know in few minutes
Can you help me with this following error:
[INFO ] 2019-05-09 11:50:25.966 [[squidlogs]>worker0] elasticsearch - retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
That typically means you are running out of disk space and have filled > 95% of disk, See the docs for how to rectify.
What I just tried was I've executed pipelines.yml and a new index was created named "logstash_apaachelogs" and then I manually deleted it using management tab. Now when I updated few logs in that location then automatically new index was created with above name.
Thanks will manage the space
What I just tried was I've executed pipelines.yml and a new index was created named "logstash_apaachelogs" and then I manually deleted it using management tab. Now when I updated few logs in that location then automatically new index was created with above name.
While using the ILM I am getting this error:
{
"indices" : {
"logstash_apchelogs-2019.05" : {
"index" : "logstash_apchelogs-2019.05",
"managed" : true,
"policy" : "IndexManagement",
"lifecycle_date_millis" : 1557400555402,
"phase" : "hot",
"phase_time_millis" : 1557400854567,
"action" : "rollover",
"action_time_millis" : 1557400854567,
"step" : "ERROR",
"step_time_millis" : 1557401400805,
"failed_step" : "check-rollover-ready",
"step_info" : {
"type" : "illegal_argument_exception",
"reason" : "index name [logstash_apchelogs-2019.05] does not match pattern '^.*-\d+'",
"stack_trace" : """
java.lang.IllegalArgumentException: index name [logstash_apchelogs-2019.05] does not match pattern '^.*-\d+'
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.generateRolloverIndexName(TransportRolloverAction.java:237)
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:120)
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:67)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:124)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:211)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:140)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:208)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:164)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:141)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:97)
at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:161)
at org.elasticsearch.xpack.indexlifecycle.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:55)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1269)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1777)
at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:115)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:128)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:244)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:199)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:164)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:192)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"""
},
"phase_execution" : {
"policy" : "IndexManagement",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d"
}
}
},
"version" : 2,
"modified_date_in_millis" : 1557393705856
}
}
}
}
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.