Cannot run curator from cron in docker container as the elasticsearch user

Elastic Stack Elasticsearch
1

In my 8.8.2 Elasticsearch container, I have this cron job in the cron table:

* 1 * * * su elasticsearch /bin/bash -c "export ES_CURATOR_USERNAME=curator ES_CURATOR_PASSWORD=xxxxx; /usr/local/bin/curator --config /config/curator-config.yml /config/curator-actions.yml"

In ES 7.17.9, this would work well to age off indices every evening at 1 a.m.

However, now when this runs I get:

Traceback (most recent call last):
  File "/usr/local/bin/curator", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 316, in cli
    set_logging(check_logging_config(
  File "/usr/local/lib/python3.8/dist-packages/curator/config_utils.py", line 51, in set_logging
    loginfo = LogInfo(log_opts)
  File "/usr/local/lib/python3.8/dist-packages/curator/logtools.py", line 136, in __init__
    self.handler = logging.FileHandler('/proc/1/fd/1')
  File "/usr/lib/python3.8/logging/__init__.py", line 1147, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.8/logging/__init__.py", line 1176, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/proc/1/fd/1'

I have found that if this command runs as the Linux root user (same curator username and password) it seems to run fine (although the output is owned by the root user and not the elasticsearch user).

Are we now required to use the root user for curator actions in crontab if the output is redirected to a file? We have used the elasticsearch user for years to run cron jobs.

2

The code was added to enable Docker container logging (i.e. you can get the logs by calling docker logs CONTAINERNAME). The code I am putting in place right now should catch edge cases like this.

With that said, I can't say I recommend containers being used this way (as though they are a VM, with cron and all of the trimmings). Containers should be stateless. Just because a thing can be done doesn't mean it should. I should not have to specially code to fit an atypical container use case.

3

The thing is that we've used Elasticsearch for at least six years, first with version 6.7.3 and then with version 7.17.9. I did not set up these cron jobs, but they have always worked until we upgraded again to 8.8.2. And since there already was a bug report for this, it is likely not an atypical case.

We are using the official Elasticsearch docker image. Can you point me to where in the Elasticsearch documentation where it says that you should not run cron inside the ES docker container?

Another issue that I would have is that my elasticsearch user does not exist outside my docker container, so I'd have to run curator as root or another user.

4

What I'm saying is not particular to the Elasticsearch docker image, but containerization in general. The idea is that the container should be run as stateless as possible (e.g. Kubernetes). The more stateful you make a container, the more things can go sideways. The most "state" a container should have to deal with is persistent storage, and that to the extent that another container should be able to just start up and point at that persistent storage and do its one thing. Modifying an existing container image to add Curator instead of just running Curator as a separate, single-purpose container that connects to the first via an API call is the part that confuses me. Regardless, I've coded around this and it will be released later today as Curator 8.0.9.

1 Like
5

Sorry for the delayed release of Curator 8.0.9, which was just published. Please let me know if you still encounter the issue with permissions writing to /proc/1/fd/1 in Docker. There may be a message or error if it attempts and fails, but it should work regardless.

6

Thanks. I'm in the middle of upgrading our code to ES/Kibana 8.11.4 so I won't be able to test this until next week.

7

I was able to retest this today and unfortunately, it still fails in the same manner.

root@elasticsearch-aln-nbadev4:/usr/share/elasticsearch# pip install elasticsearch_curator==8.0.9
Requirement already satisfied: elasticsearch_curator==8.0.9 in /usr/local/lib/python3.8/dist-packages (8.0.9)
Requirement already satisfied: es-client==8.12.3 in /usr/local/lib/python3.8/dist-packages (from elasticsearch_curator==8.0.9) (8.12.3)
Requirement already satisfied: six>=1.16.0 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (1.16.0)
Requirement already satisfied: elasticsearch8==8.12.0 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (8.12.0)
Requirement already satisfied: voluptuous>=0.14.1 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (0.14.1)
Requirement already satisfied: ecs-logging==2.1.0 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (2.1.0)
Requirement already satisfied: pyyaml==6.0.1 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (6.0.1)
Requirement already satisfied: click==8.1.7 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (8.1.7)
Requirement already satisfied: certifi>=2023.11.17 in /usr/local/lib/python3.8/dist-packages (from es-client==8.12.3->elasticsearch_curator==8.0.9) (2023.11.17)
Requirement already satisfied: elastic-transport<9,>=8 in /usr/local/lib/python3.8/dist-packages (from elasticsearch8==8.12.0->es-client==8.12.3->elasticsearch_curator==8.0.9) (8.12.0)
Requirement already satisfied: urllib3<3,>=1.26.2 in /usr/local/lib/python3.8/dist-packages (from elastic-transport<9,>=8->elasticsearch8==8.12.0->es-client==8.12.3->elasticsearch_curator==8.0.9) (2.2.0)
root@elasticsearch-aln-nbadev4:/usr/share/elasticsearch# su elasticsearch /bin/bash -c "export ES_CURATOR_USERNAME=curator ES_CURATOR_PASSWORD=xxxxx; /usr/local/bin/curator --config /config/curator-config.yml /config/curator-actions.yml"
Traceback (most recent call last):
  File "/usr/local/bin/curator", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 249, in cli
    configure_logging(cfg, ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/es_client/helpers/logging.py", line 175, in configure_logging
    set_logging(check_logging_config({'logging': logcfg}))
  File "/usr/local/lib/python3.8/dist-packages/es_client/helpers/logging.py", line 274, in set_logging
    loginfo = LogInfo(log_opts)
  File "/usr/local/lib/python3.8/dist-packages/es_client/helpers/logging.py", line 56, in __init__
    with open(fpath, 'wb+', buffering=0) as fhdl:
PermissionError: [Errno 13] Permission denied: '/proc/1/fd/1'
8

I see what I need to do. The try/except block needs to be out another level. Let me push an 8.0.10 out soon.

1 Like
9

8.0.10 is out. Please try again.

10

Another user reported that the 8.0.10 fix worked for their Docker use case, so I'm curious to hear yours.

11

Yes, I can confirm that curator version 8.0.10 fixed the issue. Thank you for the quick fix.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.