Cannot run curator from cron in docker container as the elasticsearch user

silentfilm · January 29, 2024, 10:18pm

In my 8.8.2 Elasticsearch container, I have this cron job in the cron table:

* 1 * * * su elasticsearch /bin/bash -c "export ES_CURATOR_USERNAME=curator ES_CURATOR_PASSWORD=xxxxx; /usr/local/bin/curator --config /config/curator-config.yml /config/curator-actions.yml"

In ES 7.17.9, this would work well to age off indices every evening at 1 a.m.

However, now when this runs I get:

Traceback (most recent call last):
  File "/usr/local/bin/curator", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 316, in cli
    set_logging(check_logging_config(
  File "/usr/local/lib/python3.8/dist-packages/curator/config_utils.py", line 51, in set_logging
    loginfo = LogInfo(log_opts)
  File "/usr/local/lib/python3.8/dist-packages/curator/logtools.py", line 136, in __init__
    self.handler = logging.FileHandler('/proc/1/fd/1')
  File "/usr/lib/python3.8/logging/__init__.py", line 1147, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.8/logging/__init__.py", line 1176, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/proc/1/fd/1'

I have found that if this command runs as the Linux root user (same curator username and password) it seems to run fine (although the output is owned by the root user and not the elasticsearch user).

Are we now required to use the root user for curator actions in crontab if the output is redirected to a file? We have used the elasticsearch user for years to run cron jobs.

theuntergeek · January 30, 2024, 12:33am

github.com/elastic/curator

Curator running failed in docker when upgrade to 8.0.8 as permssion error

opened 03:53AM - 13 Oct 23 UTC

Jae1313

## For usage questions and help Please create a topic at https://discuss.elasti…c.co/c/elasticsearch Perhaps a topic there already has an answer for you! ## To submit a bug or report an issue ### Expected Behavior ### Actual Behavior ### Steps to Reproduce the Problem 1. install curator in docker, run the curator as elasticsearch user instead of root user command: su elasticsearch /bin/bash -c "/usr/local/bin/curator --config /config/curator-config.yml /config/curator-actions.yml" curator-config.yml example: elasticsearch: client: hosts: - https://aln-tcdvc7vm-69.labs.netscout.com:9200 cloud_id: ca_certs: /config/certs/cacert.pem client_cert: client_key: verify_certs: request_timeout: 30 other_settings: master_only: False username: password: api_key: id: api_key: token: logging: loglevel: INFO logfile: /logs/es-curator-actions.log logformat: default blacklist: ['elastic_transport', 'urllib3'] Command result as following error: Traceback (most recent call last): File "/usr/local/bin/curator", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 34, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 316, in cli set_logging(check_logging_config( File "/usr/local/lib/python3.8/dist-packages/curator/config_utils.py", line 51, in set_logging loginfo = LogInfo(log_opts) File "/usr/local/lib/python3.8/dist-packages/curator/logtools.py", line 136, in __init__ self.handler = logging.FileHandler('/proc/1/fd/1') File "/usr/lib/python3.8/logging/__init__.py", line 1147, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python3.8/logging/__init__.py", line 1176, in _open return open(self.baseFilename, self.mode, encoding=self.encoding) PermissionError: [Errno 13] Permission denied: '/proc/1/fd/1' ### Specifications - Version: curator 8.0.8 - Platform: elasticsearch 8.8.2 docker version with Ubuntu 20.04.6 LTS - Subsystem: ## Context (Environment) ## Detailed Description

The code was added to enable Docker container logging (i.e. you can get the logs by calling docker logs CONTAINERNAME). The code I am putting in place right now should catch edge cases like this.

With that said, I can't say I recommend containers being used this way (as though they are a VM, with cron and all of the trimmings). Containers should be stateless. Just because a thing can be done doesn't mean it should. I should not have to specially code to fit an atypical container use case.

silentfilm · January 30, 2024, 3:15pm

The thing is that we've used Elasticsearch for at least six years, first with version 6.7.3 and then with version 7.17.9. I did not set up these cron jobs, but they have always worked until we upgraded again to 8.8.2. And since there already was a bug report for this, it is likely not an atypical case.

We are using the official Elasticsearch docker image. Can you point me to where in the Elasticsearch documentation where it says that you should not run cron inside the ES docker container?

Another issue that I would have is that my elasticsearch user does not exist outside my docker container, so I'd have to run curator as root or another user.

theuntergeek · January 30, 2024, 3:51pm

What I'm saying is not particular to the Elasticsearch docker image, but containerization in general. The idea is that the container should be run as stateless as possible (e.g. Kubernetes). The more stateful you make a container, the more things can go sideways. The most "state" a container should have to deal with is persistent storage, and that to the extent that another container should be able to just start up and point at that persistent storage and do its one thing. Modifying an existing container image to add Curator instead of just running Curator as a separate, single-purpose container that connects to the first via an API call is the part that confuses me. Regardless, I've coded around this and it will be released later today as Curator 8.0.9.

theuntergeek · February 1, 2024, 7:32am

Sorry for the delayed release of Curator 8.0.9, which was just published. Please let me know if you still encounter the issue with permissions writing to /proc/1/fd/1 in Docker. There may be a message or error if it attempts and fails, but it should work regardless.

silentfilm · February 1, 2024, 2:30pm

Thanks. I'm in the middle of upgrading our code to ES/Kibana 8.11.4 so I won't be able to test this until next week.

silentfilm · February 1, 2024, 9:21pm

I was able to retest this today and unfortunately, it still fails in the same manner.

root@elasticsearch-aln-nbadev4:/usr/share/elasticsearch# pip install elasticsearch_curator==8.0.9
Requirement already satisfied: elasticsearch_curator==8.0.9 in /usr/local/lib/python3.8/dist-packages (8.0.9)
Requirement already satisfied: es-client==8.12.3 in /usr/local/lib/python3.8/dist-packages (from elasticsearch_curator==8.0.9) (8.12.3)
Requirement already satisfied: six>=1.16.0 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (1.16.0)
Requirement already satisfied: elasticsearch8==8.12.0 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (8.12.0)
Requirement already satisfied: voluptuous>=0.14.1 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (0.14.1)
Requirement already satisfied: ecs-logging==2.1.0 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (2.1.0)
Requirement already satisfied: pyyaml==6.0.1 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (6.0.1)
Requirement already satisfied: click==8.1.7 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (8.1.7)
Requirement already satisfied: certifi>=2023.11.17 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (2023.11.17)
Requirement already satisfied: elastic-transport<9,>=8 in /usr/local/lib/python3.8/dist-packages (from elasticsearch8==8.12.0->es-client==8.12.3->elasticsearch_curator==8.0.9) (8.12.0)
Requirement already satisfied: urllib3<3,>=1.26.2 in /usr/local/lib/python3.8/dist-packages (from elastic-transport<9,>=8->elasticsearch8==8.12.0->es-client==8.12.3->elasticsearch_curator==8.0.9) (2.2.0)
root@elasticsearch-aln-nbadev4:/usr/share/elasticsearch# su elasticsearch /bin/bash -c "export ES_CURATOR_USERNAME=curator ES_CURATOR_PASSWORD=xxxxx; /usr/local/bin/curator --config /config/curator-config.yml /config/curator-actions.yml"
Traceback (most recent call last):
  File "/usr/local/bin/curator", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 249, in cli
    configure_logging(cfg, ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/es_client/helpers/logging.py", line 175, in configure_logging
    set_logging(check_logging_config({'logging': logcfg}))
  File "/usr/local/lib/python3.8/dist-packages/es_client/helpers/logging.py", line 274, in set_logging
    loginfo = LogInfo(log_opts)
  File "/usr/local/lib/python3.8/dist-packages/es_client/helpers/logging.py", line 56, in __init__
    with open(fpath, 'wb+', buffering=0) as fhdl:
PermissionError: [Errno 13] Permission denied: '/proc/1/fd/1'

theuntergeek · February 2, 2024, 1:01am

I see what I need to do. The try/except block needs to be out another level. Let me push an 8.0.10 out soon.

theuntergeek · February 2, 2024, 4:23am

8.0.10 is out. Please try again.

theuntergeek · February 2, 2024, 4:07pm

Another user reported that the 8.0.10 fix worked for their Docker use case, so I'm curious to hear yours.

silentfilm · February 2, 2024, 4:08pm

Yes, I can confirm that curator version 8.0.10 fixed the issue. Thank you for the quick fix.

system · March 1, 2024, 4:09pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.