Config management

Elastic Community and Ecosystem
1

I'm the creator of Elastic's Puppet Modules for Elasticsearch and Logstash. As you can imagine, I care a lot about configuration management and am a Puppet fan.

I'd welcome feedback from the community on the modules and other things you think we could do to make your life better with configuration management.

2

We use Ansible so I can't comment on the Puppet module, but apart from obvious stuff like installing the software in the first place—which is easy to automate—I've had to (or will have to) write not-entirely-trivial Ansible tasks and/or modules for the following:

  • Index template deployment.
  • Dynamic cluster settings deployment.
  • Supporting rolling restarts by waiting for the cluster to become green, taking the node out of the load balancer, disabling shard allocations, restarting the service, waiting until it's up (and then some), and finally getting it back in the load balancer and reenabling shard allocations.
  • Figuring out a suitable discovery.zen.minimum_master_nodes value given that some nodes might have node.master set to false.

There are about a million Ansible roles and playbooks named after Elasticsearch on Ansible Galaxy and I haven't looked at them all, but those I have looked at seem to be focused on the stuff that's easy anyway. If I'd ask for anything it'd be automation of the rest.

3

We're working on an Ansible playbook at the moment, it's still under pretty heavy development so it's not publicly accessible for the moment, but keep your eye out for it!

4

I've used both the puppet module and the chef cookbook to setup elasticsearch at my various employers.

Its been a while since I looked at the puppet module, but I remember that the chef cookbook had the following features that were very nice:

  • Support optionally configuring nignx reverse proxy for authentication
  • Prevent chef converges from triggering restarts of elasticsearch service.
  • Validate downloads with checksums.
  • Install multiple versions simultaneously and symlink the newest one to 'current' (easy rollback)
  • Install plugins easily
  • Generate elasticsearch.yml config file dynamically by passing a hash of values to a resource. (No need to have all config values defined in the .erb template)

The puppet module likely supported some of these, but not all.

5

Hi Spuder,

Most of the topics you noted are indeed supported by the Puppet module.
The only things that are not included are the nginx proxy and multiple versions.

The nginx proxy is not included because it should not be part of the module.
The ES module is focused on a single thing which is ES. There are allot of other modules that can handle proxying.

Regarding multiple versions it gets a bit harder since i handle installation using packages :smile:

Hope this explains it a bit for the puppet module.

6

The dependency on file_concat is somewhat painful, as the errors thrown are very non-descriptive (ie sourcing a file that doesn't exist). Please consider switching to something like http://github.com/ripienaar/puppet-concat/. This also removes the need for special types/providers and just uses native puppet code.

+1 on the rolling restarts. Not sure if this is even (cleanly) possible with puppet; sounds like an orchestration problem (ansible / mcollective?).

Thanks for your work on the puppet modules.

7

Hi @pieterl,

puppetlabs-concat is much slower then the library implementation i have.
However they are working on 2.0 which will be using my version under the hood.
I might consider moving to it at some point.

Ps. file concat has been improved a lot over the past weeks.

8

I have just recently released some Ansible roles/plays and etc around highly available solutions. You can read more at the link below but I am looking for some valuable input on additional functionality and adjustments that may need to be made. Feel free to ping me by all means.

And my GitHub repo can be found here

9

Hi @electrical,

I use several of your Puppet modules 'elasticsearch' and 'logstash' and want to thank you for your job :smile:

10

Hi Mark.
Very interested to hear more about your experiences with Ansible! We recently started using Ansible and find it quite easy to work with. Any news on when/if you can share the playbook?

11

We're using Puppet together with Vagrant to spin up an ELK box running as a virtual machine locally. We use this a lot for testing and developing, and it works great! Would perhaps be nice to have a Kibana puppet module.
The vagrant-elk-box can be found here

12

Not sure, but I'll ask :slight_smile:

13

Its on the roadmap to build an official Puppet module for Kibana. No ETA yet though. will keep the tread updated.