Curator returns empty output for elasticsearch user

George_Wainwright · February 12, 2019, 8:24pm

On my Ubuntu Xenial host curator doesn't work at all under elasticsearch user:
root@elastic1:~# su - elasticsearch -s /bin/bash
elasticsearch@elastic1:~ export LC_ALL=C.UTF-8; export LANG=C.UTF-8 elasticsearch@elastic1:~ curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml
elasticsearch@elastic1:~$

But it works under root user by exporting those locales or if I do the next:

root@elastic1:~# sudo -u elasticsearch export LC_ALL=C.UTF-8; export LANG=C.UTF-8; curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml

Curator version:
root@elastic1:~# curator --version
curator, version 5.6.0

How to make it work under elasticsearch user? What do I do wrong?

theuntergeek · February 12, 2019, 8:40pm

Does the Elasticsearch user have full read privileges to read from /etc/elasticsearch/curator.yml and /etc/elasticsearch/curator/actions/dns-retention.yaml?

Are there SELinux rules that would prevent that user from making outbound network calls?

George_Wainwright · February 12, 2019, 9:24pm

Does the Elasticsearch user have full read privileges to read from /etc/elasticsearch/curator.yml and /etc/elasticsearch/curator/actions/dns-retention.yaml ?
Yes, it has.

Are there SELinux rules that would prevent that user from making outbound network calls?

How to check it on Ubuntu?

theuntergeek · February 12, 2019, 9:40pm

Not an Ubuntu wizard, but in Ubuntu I believe they use app-armor instead of SELinux.

What happens if you log in as the Elasticsearch user, instead of using su?

George_Wainwright · February 13, 2019, 7:35am

elasticsearch user doesn't have login permission it is set /bin/false for it in env variables.

Like I said I can execute command via sudo -u elasticsearch export LC_ALL=C.UTF-8; export LANG=C.UTF-8; curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml

it works fine. But in fact it gives empty ouptput if I do su - elasticsearch -s /bin/bash or If I do it in crontab by setting those locale variables in cron as well.

Another fact is this is an AWS EC2 instance.

theuntergeek · February 13, 2019, 12:24pm

That’s probably because cron expects a shell to be able to fork. You’d have to run it out of the root cron like you’ve been doing.

George_Wainwright · February 13, 2019, 1:03pm

Aaron,

the problem not in cron. When I run curator --help manually under elasticsearch user it doesn't give me an output as well. The root of issue lies in something else but not in cron.

George_Wainwright · February 13, 2019, 1:25pm

Hold on I was a bit fast with answer. Once I imported locales I am able to see --help output, but command:

curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml

doesn't give any output.

theuntergeek · February 13, 2019, 1:42pm

Try dialing up the loglevel to DEBUG and see what, if anything, comes of it.

George_Wainwright · February 13, 2019, 1:58pm

It doesn't output anything to logs as well, but looks like I found the root of issue:
if I add elasticsearch into sudoers file with elasticsearch ALL=(ALL) NOPASSWD:ALL
It works under that user. So the issue lay in sudo privileges for elasticsearch user.

theuntergeek · February 13, 2019, 2:07pm

If you’re going to do that, you might as well run it as root. A user without an executable shell is probably the real culprit here. The command you are running simply executes the root user’s chosen shell as the Elasticsearch user. I will test and verify this at some point today, VPN permitting.

George_Wainwright · February 13, 2019, 2:11pm

Well, how to overcome it in order I do not add a user to sudoers and do not execute command with sudo?

theuntergeek · February 13, 2019, 3:16pm

I created 2 users on a test Ubuntu 16.04 VM I have:

has_login:x:2001:2001::/home/has_login:/bin/bash
nologin:x:2002:2002::/home/nologin:/bin/false

This is what I found.

When run as the root user with the user that has a bash shell:

# sudo -u has_login curator --config curator.yml open-test.yml
2019-02-13 14:54:45,098 INFO      Preparing Action ID: 1, "open"
2019-02-13 14:54:45,154 INFO      Trying Action ID: 1, "open": open all indices older than 0 days should only use with --dry-run as test for output
2019-02-13 14:54:45,214 INFO      Opening selected indices: ['.triggered_watches', '.monitoring-alerts-6', '.monitoring-es-6-2019.02.12', '.tasks', '.monitoring-es-6-2019.02.13', '.kibana_2', 'netflow-000002', '.monitoring-kibana-6-2019.02.13', 'netflow-000001', 'netflow-000003', '.monitoring-kibana-6-2019.02.12', '.watcher-history-9-2019.02.12', '.watches', '.watcher-history-9-2019.02.13', '.kibana_1']
2019-02-13 14:54:45,223 INFO      Action ID: 1, "open" completed.
2019-02-13 14:54:45,223 INFO      Job completed.
# su - has_login -c 'curator --config curator.yml open-test.yml'
2019-02-13 14:55:38,720 INFO      Preparing Action ID: 1, "open"
2019-02-13 14:55:38,776 INFO      Trying Action ID: 1, "open": open all indices older than 0 days should only use with --dry-run as test for output
2019-02-13 14:55:38,854 INFO      Opening selected indices: ['.kibana_2', 'netflow-000002', 'netflow-000003', '.tasks', '.watches', '.watcher-history-9-2019.02.12', '.triggered_watches', '.monitoring-alerts-6', '.monitoring-kibana-6-2019.02.13', '.monitoring-kibana-6-2019.02.12', 'netflow-000001', '.monitoring-es-6-2019.02.13', '.watcher-history-9-2019.02.13', '.monitoring-es-6-2019.02.12', '.kibana_1']
2019-02-13 14:55:38,861 INFO      Action ID: 1, "open" completed.
2019-02-13 14:55:38,861 INFO      Job completed.

When I run as the user with /bin/false as the shell:

# sudo -u nologin curator --config /bigdisk/buh/curator_packages/curator.yml /bigdisk/buh/curator_packages/open-test.yml
2019-02-13 14:57:00,627 INFO      Preparing Action ID: 1, "open"
2019-02-13 14:57:00,683 INFO      Trying Action ID: 1, "open": open all indices older than 0 days should only use with --dry-run as test for output
2019-02-13 14:57:00,743 INFO      Opening selected indices: ['.kibana_1', '.monitoring-es-6-2019.02.13', '.monitoring-kibana-6-2019.02.13', '.watcher-history-9-2019.02.13', 'netflow-000002', '.monitoring-alerts-6', '.watches', '.triggered_watches', 'netflow-000003', '.watcher-history-9-2019.02.12', '.monitoring-es-6-2019.02.12', '.tasks', '.kibana_2', '.monitoring-kibana-6-2019.02.12', 'netflow-000001']
2019-02-13 14:57:00,751 INFO      Action ID: 1, "open" completed.
2019-02-13 14:57:00,752 INFO      Job completed.

# su - nologin -c 'curator --config /bigdisk/buh/curator_packages/curator.yml /bigdisk/buh/curator_packages/open-test.yml'
# (NO OUTPUT)

In other words, the user must have a shell if you plan on using su -. If you allow your sudoers file to have a blanket "Elasticsearch user can do anything", i.e., elasticsearch ALL=(ALL) NOPASSWD:ALL, this is a security risk. It's much safer to have a plain user with an executing shell, and just let that user have a crontab entry:

# su - elasticsearch
$ crontab -l
no crontab for elasticsearch
$ crontab -e
< add 0 2 * * * /usr/bin/curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml or something like it >
$ crontab -l
0 2 * * * /usr/bin/curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml

George_Wainwright · February 13, 2019, 3:41pm

Aaron,

thank you for the investigation, but I actually set shell for that user:
chsh --shell /bin/bash elasticsearch

but is still didn't work after it.

theuntergeek · February 13, 2019, 4:08pm

So, now that the elasticsearch user has a shell, does it also have a home directory?

elasticsearch:x:1000:1000:Elasticsearch User,,,:/home/elasticsearch:/bin/bash

What happens when you su - elasticsearch?

# su - elasticsearch

$ curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.
$

What happens if you switch to the elasticsearch user and then execute?

George_Wainwright · February 13, 2019, 4:13pm

elasticsearch:x:112:116::/home/elasticsearch:/bin/bash
elasticsearch@elastic1:~ export LC_ALL=C.UTF-8; export LANG=C.UTF-8 elasticsearch@elastic1:~ curator --help
Usage: curator [OPTIONS] ACTION_FILE

Curator for Elasticsearch indices.

See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
--config PATH Path to configuration file. Default: ~/.curator/curator.yml
--dry-run Do not perform any changes.
--version Show the version and exit.
--help Show this message and exit.

elasticsearch@elastic1:~ curator --config /etc/elasticsearch/curator.yml /etc/elasticsearch/curator/actions/dns-retention.yaml elasticsearch@elastic1:~

As you can see the last command didn't give an output.

theuntergeek · February 13, 2019, 4:23pm

This suggests that the Elasticsearch user may not have read permissions to one of those files. This can be verified:

$ cat /etc/elasticsearch/curator.yml
... (output)
$ cat /etc/elasticsearch/curator/actions/dns-retention.yaml
... (output)

Barring this, as previously stated, it could be that the user cannot open a socket (also permissions).

George_Wainwright · February 13, 2019, 4:24pm

It has permissions, I am able to cat both of those files under elasticsearch user.

theuntergeek · February 13, 2019, 4:25pm

Okay. Does this execute fully when executed as root? If so, it implies that root has permission to do something that the Elasticsearch user does not.

George_Wainwright · February 13, 2019, 4:26pm

Does this execute fully when executed as root?
I removed elasticsearch from sudoers so it is not being executed as root now.

Being added to sudoers command is being executed properly.