Snapshot creation using curator 4.2

I want to create a snapshot older than 30 days indices using curator 4.2

so i created two curator configuration file :

cat ~/curator_config.yml

# Remember, leave a key empty if there is no value.  None will be a string, not a Python "NoneType"
client:
  hosts:
    - loclahost.com
  port: 9200
  timeout: 30 logging:
  loglevel: INFO

cat ~/action_file_snapshot.yml

---
# Remember, leave a key empty if there is no value.  None will be a string, not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you want to use this action as a template, be sure to set 
# this to False after copying it.
actions:
  1:
    action: snapshot
    description: >-
      Snapshot logstash- prefixed indices older than 30 day (based on index
      creation_date) with the default snapshot name pattern of
      'curator-%Y%m%d%H%M%S'.  Wait for the snapshot to complete.  Do not skip
      the repository filesystem access check.  Use the other options to create
      the snapshot.
    options:
      repository:
      # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
      name: s3-backup
      ignore_unavailable: False
      include_global_state: True
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: False
      timeout_override:
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 30
      exclude:

After this files creation, i execute this below command

 curator --config ~/curator_config.yml ~/action_file_snapshot.yml                               Traceback (most recent call last):                                                                                      
  File "/usr/bin/curator", line 9, in <module>
    load_entry_point('elasticsearch-curator==4.2.6', 'console_scripts', 'curator')()
  File "/usr/lib/python2.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/curator/cli.py", line 115, in cli
    client_args = process_config(config)
  File "/usr/lib/python2.6/site-packages/curator/config_utils.py", line 46, in process_config
    config = test_config(yaml_file)
  File "/usr/lib/python2.6/site-packages/curator/config_utils.py", line 9, in test_config
    yaml_config  = get_yaml(config)
  File "/usr/lib/python2.6/site-packages/curator/utils.py", line 56, in get_yaml
    'Unable to parse YAML file. Error: {0}'.format(e))
curator.exceptions.ConfigurationError: Unable to parse YAML file. Error: mapping values are not allowed here
  in "<string>", line 6, column 22:
      timeout: 30 logging:

I am using curator 4.2 version and this commands execute in elasticsearch master node

$curator --version
curator, version 4.2.6

kindly help me this configuration are correct or not, if wrong kindly correct this configuration files.

The error:

Is explained by the configuration here:

logging: should be on its own line, and should be a root-level key in the yaml file.

ok thank you, now i tried take a snapshot using curator 3.5.1, its working fine. wait i will update you.

Now i execute following command but ERROR No indices provided WARNING Job did not complete successfully.

]# curator --host localhost.com snapshot --repository s3-backup indices --time-unit days --older-than 30 --timestring '%Y.%m.%d' --prefix topbeat-
2017-02-21 23:33:47,860 INFO Job starting: snapshot indices
2017-02-21 23:33:47,860 INFO Overriding default connection timeout for snapshot action. New timeout: 21600
2017-02-21 23:33:48,099 INFO Action snapshot will be performed on the following indices: [u'topbeat-2016.12.07', u'topbeat-2016.12.09', u'topbeat-2016.12.10', u'topbeat-2016.12.11', u'topbeat-2016.12.12', u'topbeat-2016.12.15', u'topbeat-2016.12.16', u'topbeat-2016.12.17', u'topbeat-2016.12.18', u'topbeat-2016.12.19', u'topbeat-2016.12.20', u'topbeat-2016.12.21', u'topbeat-2016.12.22', u'topbeat-2016.12.23', u'topbeat-2016.12.24', u'topbeat-2016.12.25', u'topbeat-2016.12.26', u'topbeat-2016.12.27', u'topbeat-2016.12.28', u'topbeat-2016.12.29', u'topbeat-2016.12.30', u'topbeat-2016.12.31', u'topbeat-2017.01.01', u'topbeat-2017.01.02', u'topbeat-2017.01.03', u'topbeat-2017.01.04', u'topbeat-2017.01.05', u'topbeat-2017.01.06', u'topbeat-2017.01.07', u'topbeat-2017.01.08', u'topbeat-2017.01.09', u'topbeat-2017.01.10', u'topbeat-2017.01.11', u'topbeat-2017.01.12', u'topbeat-2017.01.13', u'topbeat-2017.01.14', u'topbeat-2017.01.15', u'topbeat-2017.01.16', u'topbeat-2017.01.17', u'topbeat-2017.01.18', u'topbeat-2017.01.19', u'topbeat-2017.01.20', u'topbeat-2017.01.21', u'topbeat-2017.01.22', u'topbeat-2017.01.23']
2017-02-21 23:33:48,273 ERROR No indices provided.
2017-02-21 23:33:48,273 WARNING Job did not complete successfully.

kindly help me

How did you get from Curator 4.2 to 3.5.1? That's a big regression that you shouldn't be doing unless you're using AWS ES versions 1.5 or 2.3.

We are using using curator 3.5.1, Elasticsearch version 2.3.1.

My first recommendation is to upgrade Curator to 4.2.6, unless you are using AWS ES. Is that the case?

HI,

I am using Curator version 3.5.1 and we are trying to set this up on an old Elasticsearch Version 1.1.2.

When I run the following (with --dry-run): curator --dry-run --host 127.0.0.1 snapshot --repository ElasticSearch2-Repo-S3 indices --older-than 53 --time-unit days --timestring '%Y.%m.%d'

It lists several Indices it will Snapshot, However when I run it without --dry-run it fails withthe following:

2017-03-07 15:54:48,815 INFO Job starting: snapshot indices
2017-03-07 15:54:48,816 INFO Overriding default connection timeout for snapshot action. New timeout: 21600
2017-03-07 15:54:48,902 INFO Action snapshot will be performed on the following indices: [u'applogs-2017.01.13', u'latencymetrics-2017.01.13', u'preadvicemetrics-2017.01.13']
2017-03-07 15:54:48,917 ERROR No indices provided.
2017-03-07 15:54:48,917 WARNING Job did not complete successfully.

@Robsi Please open a new topic as your issue is version specific, and this topic is specific to Curator 4.2

I have created a new ticket - Curator 3.5.1 Snapshot issue

Could you take alook please

Hi,

Now i installed curator 4.2,

Action file

Remember, leave a key empty if there is no value. None will be a string,

not a Python "NoneType"

Also remember that all examples have 'disable_action' set to True. If you

want to use this action as a template, be sure to set this to False after

copying it.

actions:
1:
action: snapshot
description: >-
Snapshot logstash- prefixed indices older than 1 day (based on index
creation_date) with the default snapshot name pattern of
'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip
the repository filesystem access check. Use the other options to create
the snapshot.
options:
repository: s3_curator
# Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
name:
ignore_unavailable: False
include_global_state: True
partial: False
wait_for_completion: True
skip_repo_fs_check: False
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 5
exclude:

config file

Remember, leave a key empty if there is no value. None will be a string,

not a Python "NoneType"

client:
hosts:
- elastic.com
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile:
logformat: default
blacklist: []

when i run this command

$ curator --config config.yml action.yml

thie error came

2017-03-28 06:15:06,865 INFO Preparing Action ID: 1, "snapshot"
2017-03-28 06:15:06,875 WARNING GET / [status:401 request:0.006s]
Unable to create client connection to Elasticsearch. Error: TransportError(401, 'security_exception', 'missing authentication token for REST request [/]')

kindly help me

That error suggests you have a username and password protected Elasticsearch, and you aren't including them

i have one doubt Where to give username password ? in this configuration file or running time .

which server username password ? server or client.

kindly it will help full to me

In order to log in to Elasticsearch, you have to provide a username and password, yes?

In your curator.yml file, that information goes in the http_auth line as follows:

http_auth: USER:PASS

And the username and password are connected by a colon : character.

1 Like

Hi Sir,

I provided a username and password like this

http_auth:
now response happened with error, kindly give a solution of this also

Config.yml:

client:
  hosts:
    - elas.com
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: REDACTED
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['urllib3']

error output:

When i run this command:

$curator --config config.yml action.yml

tbngmanualtest-catalina-log-2017.03.24,tbng-tbngmanualtest-json-log-2017.03.21,tbng-tbngmanualtest-json-log-2017.03.22,tbng-tbngmanualtest-json-log-2017.03.27,tbng-tbngmanualtest-json-log-2017.03.29,topbeat-2017.03.04,topbeat-2017.03.08,topbeat-2017.03.26,topbeat-2017.03.28,undefined-logs-2017.02.14,undefined-logs-2017.02.17,undefined-logs-2017.02.26,undefined-logs-2017.02.27,undefined-logs-2017.03.02,undefined-logs-2017.03.03/_stats/store,docs [status:N/A request:0.002s]
Traceback (most recent call last):
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 386, in _make_request
  File "<string>", line 2, in raise_from
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 382, in _make_request
  File "/usr/local/lib/python3.5/http/client.py", line 1197, in getresponse
  File "/usr/local/lib/python3.5/http/client.py", line 297, in begin
  File "/usr/local/lib/python3.5/http/client.py", line 266, in _read_status
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buh/.local/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 95, in perform_request
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 649, in urlopen
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 324, in increment
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/packages/six.py", line 685, in reraise
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 386, in _make_request
  File "<string>", line 2, in raise_from
  File "/home/buh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 382, in _make_request
  File "/usr/local/lib/python3.5/http/client.py", line 1197, in getresponse
  File "/usr/local/lib/python3.5/http/client.py", line 297, in begin
  File "/usr/local/lib/python3.5/http/client.py", line 266, in _read_status
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

You probably didn't intend to include the actual username and password (if those were), so I edited your post and redacted that portion.

Please encapsulate your code in triple backticks:

```
CODE
```

It makes the output much more readable.

http.client.RemoteDisconnected: Remote end closed connection without response

and

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

Both of these suggest your Elasticsearch endpoint may be behind a load balancer, a proxy, or some other gateway. Is that the case? These errors are highly atypical, and those are the only times I can expect to receive such an error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.