Elasticsearch fails to get installed when Elasticsearch Puppet module is upgraded to 0.13.2

Elastic Stack Elasticsearch
1

@tylerjl
I upgraded elasticsearch puppet module from 0.10.3 to 0.13.2. When I run puppet apply using the below command, elasticsearch fails to get installed.

puppet apply /etc/puppet/environments/dbe_metrics/manifests/site.pp --environment=dbe_metrics --debug

The error that I get is:

Debug: Puppet::Type::Elasticsearch_plugin::ProviderElasticsearch_plugin: file /usr/share/elasticsearch/bin/elasticsearch-plugin does not exist
Debug: Creating default schedules

I've uploaded the following files to gist:

  1. profile elasticsearch_cluster.pp that installs elasticsearch and also installs plugins
  2. the role es_logstash.pp that is invoked when the main manifest site.pp is called
  3. ndd_common.yaml - hieradata file
  4. puppet-apply-debug.txt - debug run of puppet apply that contains full log
  5. site.pp

My code runs quite fine with Elasticsearch puppet module 0.10.3.

Any pointers?

2

Hey @sandeepkanabar! I'll try and explain what's happening here:

Because the module manages plugins as native puppet types, at the beginning of each puppet run (before it takes any actions), puppet will try and enumerate existing "instances" for native types that may exist on the system. It figures out what native types to look for by checking that executables or files exist, and skips checks for those that don't. What you're seeing is this pre-run stage look for the plugin executables and fail to find them, so it doesn't do the initial check to see what plugins are installed before managing them.

To demonstrate, I ran a dummy manifest on a clean CentOS 6 VM and got similar output, but later on in the puppet run (once those executables are installed) the run continues just fine and calls the plugin command:

Notice: Compiled catalog for localhost.local in environment production in 1.43 seconds
Debug: Puppet::Type::Elasticsearch_plugin::ProviderElasticsearch_plugin: file /usr/share/elasticsearch/bin/elasticsearch-plugin does not exist
Debug: Puppet::Type::Elasticsearch_plugin::ProviderPlugin: file /usr/share/elasticsearch/bin/plugin does not exist
Debug: Puppet::Type::Elasticsearch_plugin::ProviderElasticsearch_plugin: file /usr/share/elasticsearch/bin/elasticsearch-plugin does not exist
Debug: Puppet::Type::Elasticsearch_plugin::ProviderPlugin: file /usr/share/elasticsearch/bin/plugin does not exist
Debug: Creating default schedules

...many lines later...

Notice: /Stage[main]/Main/Elasticsearch::Plugin[lmenezes/elasticsearch-kopf]/Elasticsearch_plugin[lmenezes/elasticsearch-kopf]/ensure: created

Those types of debug messages aren't necessarily indicative of a failed run, so we probably need to look at the entirety of the logs for the puppet run to see what's happening. If Elasticsearch isn't getting installed, the error probably happens sometime after that debug message about bin/plugin missing.

3

@tylerjl

Thank you very much for the detailed explanation. Thanks for doing a quick test with dummy manifest. From the output you pasted, it seems that while your execution goes ahead after Debug: Creating default schedules, mine throws the following error messages after default schedules Line.

During my first run I got this error message Error: Could not find dependent File[/usr/share/elasticsearch/plugins] for File[/usr/share/elasticsearch/plugins/shield. During the second run, I got this error message:

Error: Could not find dependent File[/usr/share/elasticsearch/plugins] for File[/usr/share/elasticsearch/plugins/marvel-agent] at /etc/puppet/modules/elasticsearch/manifests/plugin.pp:174

That's all. There is nothing else in puppet logs. Does this give some clue?

4

Ah, I think I see where the bug is. I've filed an issue to fix this one.

In the meantime, I think you should be able to work around this issue with something like the following in your puppet manifest:

file { $::elasticsearch::plugindir:
  ensure  => directory,
  require => Class['elasticsearch::config'],
}

There may be some additional ordering parameters you may need to specify, but that's one workaround you can think of. Let me know how far this gets you.

5

Thank you very much @tylerjl. I'll try that. Thanks for filing the bug. As of now, we have decided to use the existing 0.10.3 module since we need to deliver it to customer ASAP. But once it's delivered and I've some time in hand, I'll give your suggestion a try and upgrade to the latest one.