# Purge Data base Elasticsearch

**URL:** <https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738>\
**Category:** Elasticsearch\
**Created:** [May 14, 2018, 12:23pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738 "2018-05-14T12:23:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 14, 2018, 12:23pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/1 "2018-05-14T12:23:31Z")

</div>

Hi,

Pleaaase, can someone tell me, how to purge the Elasticsearch database automatically every week ???

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:30am UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/2 "2018-05-15T00:30:36Z")

</div>

Elasticsearch can not do that automatically, on it's own, just yet. You need to schedule an external job that does this for you. One tool that you can that helps writting this job is [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html)

This article is also a good one that shows how to run it in AWS: [https://www.elastic.co/blog/serverless-elasticsearch-curator-on-aws-lambda](https://www.elastic.co/blog/serverless-elasticsearch-curator-on-aws-lambda)

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 7:36am UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/3 "2018-05-15T07:36:21Z")

</div>

I have a problem, I was sending logs from a server using Filebeat to my server. Once I set up the curator and I made the command : curator --config ..... ,The other server stopped sending the logs. Can you help me solve this problem ?

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 11:44am UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/4 "2018-05-15T11:44:04Z")

</div>

Did you execute it with an `action.yml` configuration? If so, attach this file here (if you are going to paste it's content make sure you do it between ```)

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:01pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/5 "2018-05-15T12:01:11Z")

</div>

' # Remember, leave a key empty if there is no value. None will be a string,

# not a Python "NoneType"

# 

# Also remember that all examples have 'disable\_action' set to True. If you

# want to use this action as a template, be sure to set this to False after

# copying it.

actions:  
1:  
action: delete\_indices  
description: \>-  
Delete indices older than 7 days (based on index name), for logstash-  
prefixed indices. Ignore the error if the filter does not result in an  
actionable list of indices (ignore\_empty\_list) and exit cleanly.  
options:  
ignore\_empty\_list: True  
timeout\_override:  
continue\_if\_exception: False  
disable\_action: True  
filters:  
- filtertype: pattern  
kind: prefix  
value: logstash-  
exclude:  
- filtertype: age  
source: name  
direction: older  
timestring: '%Y.%m.%d'  
unit: days  
unit\_count: 7  
exclude: '

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:03pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/6 "2018-05-15T12:03:33Z")

</div>

It is very difficult to read yaml like this. Please post it between a pair of "```" (it's 3x `)

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:08pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/7 "2018-05-15T12:08:16Z")

</div>

```auto
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude: 

```

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:16pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/8 "2018-05-15T12:16:06Z")

</div>

The problem is, it change my logstash, I can't start it as a service anymore !! I don't know why, everything changes.

Everythnig was working before I just install curator and rename my conf file in logstash and everythnig goes wrong ...

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [May 15, 2018, 12:16pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/9 "2018-05-15T12:16:27Z")

</div>

I put this in preformatted tags, as the triple back ticks were not cutting it, for some reason.

This action file won't work, because (as it says in the comment at the top), you have `disable_action: True`, which will prevent it from even running.

Also, this file does not need the comments, and it would be helpful if it started with `---`, which indicates YAML.

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:22pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/10 "2018-05-15T12:22:34Z")

</div>

I unistall curator, and my logstash still blocked. He is not listening and I have java erros !!

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:24pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/11 "2018-05-15T12:24:00Z")

</div>

What are the errors?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [May 15, 2018, 12:24pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/12 "2018-05-15T12:24:24Z")

</div>

You have done something else. Curator is not the cause of this. In your other post, you mention that you changed your logstash conf file name. This is probably what caused this, since it's not possible for Curator to have done it.

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:27pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/13 "2018-05-15T12:27:36Z")

</div>

````auto
May 15 11:36:01 frghcslnetv10 systemd[1]: Starting logstash...
May 15 11:36:01 frghcslnetv10 polkitd[705]: Unregistered Authentication Agent for unix-process:2635:110069211 (system bus name :1.4650, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, loca
May 15 11:36:27 frghcslnetv10 systemd-journal[503]: Suppressed 12 messages from /system.slice/logstash.service
May 15 11:36:27 frghcslnetv10 logstash[2641]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
May 15 11:36:28 frghcslnetv10 logstash[2641]: 2018-05-15 11:36:28,335 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plai
May 15 11:36:28 frghcslnetv10 logstash[2641]: at java.io.FileOutputStream.open0(Native Method)
May 15 11:36:28 frghcslnetv10 logstash[2641]: at java.io.FileOutputStream.open(Unknown Source)
May 15 11:36:28 frghcslnetv10 logstash[2641]: at java.io.FileOutputStream.<init>(Unknown Source)
May 15 11:36:28 frghcslnetv10 logstash[2641]: at java.io.FileOutputStream.<init>(Unknown Source) ```

[2018-05-15T11:51:29,521][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-05-15T11:51:29,522][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-05-15T11:51:29,541][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-05-15T11:51:29,568][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-05-15T11:51:29,568][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-05-15T11:51:29,569][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-05-15T11:51:29,570][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-05-15T11:51:29,586][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-05-15T11:51:29,970][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-05-15T11:51:29,977][INFO][org.logstash.beats.Server] Starting server on port: 5044
[2018-05-15T11:51:29,979][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6bcbfd4c@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}
[2018-05-15T11:51:29,980][INFO][logstash.agent] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-05-15T11:52:02,301][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<SystemCallError: Unknown error (SystemCallError) - <STDOUT>>, :backtrace=>["org/jruby/RubyIO.java:1457:in `write'", "org/jruby/RubyIO.java:1428:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-stdout-3.1.4/lib/logstash/outputs/stdout.rb:43:in `block in multi_receive_encoded'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-stdout-3.1.4/lib/logstash/outputs/stdout.rb:42:in `multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:90:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:15:in `block in multi_receive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:477:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:476:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:428:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:386:in `block in start_workers'"]}
[2018-05-15T11:52:02,525][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit ```
````

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:29pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/14 "2018-05-15T12:29:16Z")

</div>

Please, always remember to format your logs, configs, etc. I kindly ask you to edit or repost with preformatted tags, as the triple ticks.

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:31pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/15 "2018-05-15T12:31:13Z")

</div>

Sorry 😊 I edit it

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:37pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/16 "2018-05-15T12:37:51Z")

</div>

It seems that Logstash can not write to `/var/log/logstash` anymore. Does it still exists?

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:39pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/17 "2018-05-15T12:39:16Z")

</div>

Yes it still exists !!

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:41pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/18 "2018-05-15T12:41:26Z")

</div>

The error seems to be coming from a `stdout` output plugin. If there is one, try removing it.

---

<div class="post-metadata">

**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\
**Post date:** [May 15, 2018, 12:51pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/19 "2018-05-15T12:51:04Z")

</div>

I thought the same I comment it, but I still have the same errors. Logstash can't start as a service, he start only with : config.reload.automatic !!

---

<div class="post-metadata">

**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\
**Post date:** [May 15, 2018, 12:53pm UTC](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738/20 "2018-05-15T12:53:19Z")

</div>

It seems it is not being able to write to `/var/log/logstash/logstash-plain.log`. Check file and directory ownership (it should be owned by `logstash` user)

[Next page](https://discuss.elastic.co/t/purge-data-base-elasticsearch/131738.md?page=2)