Elasticsearch Group set to undef while creating instance dirs

[sandeepkanabar]

In instance.pp, when creating instance data_dir, log_dir and config_dir, while the user is set to elasticsearch, the group is set to undef. This creates the directories as elasticsearch:root. Is there any reason why the group cannot be set to elasticsearch too like group => $elasticsearch::elasticsearch_group while creating the above dirs?

While creating the ES config file elasticsearch.yml, both the owner and group are set. I believe the same should happen when directories are created. Please help me to understand if that is not the case.

[tylerjl (Tyler Langlois)]

Hi @sandeepkanabar,

First I just want to confirm that there's not any problems that the current configuration is causing for you, correct? If there's a bug causing instances to fail or directories being unreadable, I'd like to find and fix it, but it sounds like you're more interesting in making things consistent (which is also a good thing).

To answer your question, the current state of the directory permissions probably just needs to be cleaned up. With that in mind, there is a ticket in the puppet-elasticsearch repo that (correctly) notes some of the current permissions aren't 100% secure and should be changed. To your specific points:

• logdir, datadir
  • These can probably be changed to use elasticsearch:elasticsearch since the running ES process is meant to write to them, they work as-is but the change wouldn't hurt anything.
• configdir
  • Per the aforementioned ticket, we should tweak the permissions so that the ES user can read its config but not write to it.

[sandeepkanabar]

Thank you for the detailed reply @tylerjl. As it stands, it indeed prevents my ES instance from starting. Before I deploy elasticsearch, I set a custom unique uid and gid so that all the nodes use the same uid and gid values. The user and group mapped to this uid:gid are elasticsearch:elasticsearch but I find that when the puppet module deploys, it actually deploys as elasticsearch:root and not elasticsearch:elasticsearch, Since the group is set to undef, it takes root. Is it because puppet runs as root? In my profile class, I then apply a workaround and do chown without which the instance will NOT come up. So, having group set to elasticsearch would very much help to avoid the permissions issue for logdir and datadir. Thoughts?

[tylerjl (Tyler Langlois)]

If your elasticsearch group and elasticsearch user are using custom UID/GIDs, the Elasticsearch instances should still work - Puppet just sets the ownership by the human-readable name, and should get your custom UID/GID numbers automatically. For context, the acceptance tests for the module end up with the elasticsearch:root ownership you've noted here but still run successfully, so if the ES process is running with elasticsearch:elasticsearch effective UID/GIDs then things should still work, so the failures you're seeing are weirdly unexpected.

So, while everything I understand about the module suggests you shouldn't see errors with that setup, changing the group ownership wouldn't break anything and should be a safe change to make (though it may be worth debugging further to figure out why this is happening, since running with those directories as elasticsearch:root works fine in other places).

[sandeepkanabar]

@tylerjl - I need to correct one thing. When I said - ES instance wouldn't start, I was mistaken. What I meant to say was ES instance would indeed start but x-pack would return 401 unauth error even for the default user elastic:changeme. Thus, even though ES instance would start, it would be quite useless since I cannot read or write anything. Once I do chown -R elasticsearch:elasticsearch to /data/elasticsearch, it would work. I use custom data dir and log dirs. in my case, my data_dir is /data/elasticsearch/data and log_dir is /data/elasticsearch/log. The very first thing I do after ES instance comes up is to apply platinum license that we have purchased and that step will invariably fail due to unauth error and puppet would abort. Even manual access using curl will fail. Does that ring a bell? I'm sorry for saying that ES didn't start. 3 days passed since I filed that ticket and somehow it slipped my mind. ES instance does start. Just that i keep getting 401 unauth.

[tylerjl (Tyler Langlois)]

Ah, that makes more sense. I suspect, however, that the problem may lie outside of filesystem permissions if you're getting a 401 from Elasticsearch, since that indicates that Elasticsearch gets started fine, the API just can't be accessed.

What version of Elasticsearch and X-Pack are you running, and what Puppet resources are you trying to manage that fail causing Puppet to abort? It sounds like there may be some API settings on the elasticsearch class you need to set if you're running a secured installation of Elasticsearch.

[sandeepkanabar]

ELK stack version is 5.4.0. And Puppet module version is 5.2.0 released on May 5th.

The 401 auth error said that it couldn't access file based realm. Isn't that related to permission issue? I edited the file instance.pp within the manifests dir and changed group => undef to group => $elasticsearch::elasticsearch_group for file { $instance_logdir: and file { $instance_datadir: and the 401 auth error disappeared. Hence, as a workaround, rather than changing instance.pp, i did chown -R elasticsearch:elasticsearch in my puppet profile class that invokes elasticsearch puppet module. It was this that led me to suggesting you to change undef to $elasticsearch::elasticsearch_group.

Does this give you some clue, @tylerjl?

[tylerjl (Tyler Langlois)]

While 401 is permission-related, it's a response from X-Pack's security features rather than a reflection of permission state on the filesystem level. If Elasticsearch itself couldn't read or write to its data or log directories, it should have failed to start up entirely and wouldn't be sending back REST responses.

As a test of X-Pack, I wrote out an example Puppet manifest to try out X-Pack on 5.4.0 to check whether it works with the elasticsearch:root filesystem permissions:

[root@centos-7-x64 /]# cd
[root@centos-7-x64 ~]# cat test.pp
class { 'elasticsearch':
	manage_repo => true,
	repo_version => '5.x',
	java_install => true,
}

elasticsearch::instance { 'es-01' : }

Elasticsearch::Plugin { instances => 'es-01' }
elasticsearch::plugin { 'x-pack' : }
[root@centos-7-x64 ~]# systemctl is-active elasticsearch-es-01
active
[root@centos-7-x64 ~]# curl -s -u elastic:changeme localhost:9200
{
  "name" : "centos-7-x64-es-01",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "vOjBXd-1ThyEd0kTP1RhOw",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "780f8c4",
    "build_date" : "2017-04-28T17:43:27.229Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.0"
  },
  "tagline" : "You Know, for Search"
}
[root@centos-7-x64 ~]# ls -la /var/lib/elasticsearch
total 12
drwxr-x---  3 elasticsearch elasticsearch 4096 May 30 21:04 .
drwxr-xr-x 23 root          root          4096 May 30 20:59 ..
drwxr-xr-x  3 elasticsearch root          4096 May 30 21:04 es-01
[root@centos-7-x64 ~]# ls -la /var/log/elasticsearch
total 12
drwxr-xr-x 3 elasticsearch elasticsearch 4096 May 30 21:04 .
drwxr-xr-x 5 root          root          4096 May 30 20:59 ..
drwxr-xr-x 2 elasticsearch root          4096 May 30 21:04 es-01
[root@centos-7-x64 ~]#

As you can see, Elasticsearch 5.4.0 is running with X-Pack installed and those directories with root group ownership. Does any particular part of this seem out-of-whack with the setup you're running?

[sandeepkanabar]

While I'm not so sure, just one small fact to bring to your notice. I am using custom dir. Other than that, i don't think there's any difference. We have platinum subs from elastic and as such, I can file a new ticket using my corporate mail id and link it to this. That way you can officially work on it, provided you want to and we can get on webex/zoom chat. I could share my screen. I could also reproduce and show you the issue.

[sandeepkanabar]

@tylerjl - when a curl request with -u elastic:changeme comes, can you tell me what happens behind the scenes? How does it validate against a file-based realm? Not necessary that it should be against a file-based realm but I very clearly remember the log file telling me file-based realm and 401. I had rm -rf log and data dirs (i was chasing a deadline and had to make it work), thus I'm not able to provide you logs . But I'll try to reproduce and share the logs.

[tylerjl (Tyler Langlois)]

@sandeepkanabar feel free to open a support ticket if you need it, we'd be happy to help

The example curl request just uses the default username and password for x-pack which should already be present after installing the plugin, so that particular user comes from the native (REST API based) realm. X-Pack security will also validate against users in the file realm if they are present. However, those should work as expected as well - just to verify it, I ran some more tests using custom directories, with the file-based realm, and things still work:

[root@centos-7-x64 ~]# cat test.pp
class { 'elasticsearch':
        manage_repo => true,
        repo_version => '5.x',
        java_install => true,
        logdir => '/srv/es/logs',
        datadir => '/srv/es/data',
        security_plugin => 'x-pack',
}
elasticsearch::plugin { 'x-pack': instances => 'es-01' }
elasticsearch::user { 'test':
        password => 'foobar',
        roles => ['superuser'],
}
elasticsearch::instance { 'es-01': }
[root@centos-7-x64 ~]# curl -s -u test:foobar localhost:9200
{
  "name" : "centos-7-x64-es-01",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "wf2LvZmVQ9SLQISs7Z3ywg",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "780f8c4",
    "build_date" : "2017-04-28T17:43:27.229Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.0"
  },
  "tagline" : "You Know, for Search"
}
[root@centos-7-x64 ~]# ls -la /srv/es/
total 16
drwxr-xr-x 4 root          root          4096 May 31 14:48 .
drwxr-xr-x 3 root          root          4096 May 31 14:48 ..
drwxr-xr-x 3 elasticsearch elasticsearch 4096 May 31 14:57 data
drwxr-xr-x 3 elasticsearch root          4096 May 31 14:57 logs
[root@centos-7-x64 ~]# ls -la /srv/es/data/
total 12
drwxr-xr-x 3 elasticsearch elasticsearch 4096 May 31 14:57 .
drwxr-xr-x 4 root          root          4096 May 31 14:48 ..
drwxr-xr-x 3 elasticsearch root          4096 May 31 14:57 es-01
[root@centos-7-x64 ~]#

[sandeepkanabar]

@tylerjl thank you for the excellent explanation. I only wish I had preserved the logs or at least captured them before rm -rf'ing them so that could let you know the exact error message I saw. Sometime next week, when I have bandwidth, I'll file a ticket and may be we can have a session. Its always awesome interacting with you