Puppet-elasticsearch options

Elastic Stack Elasticsearch
1

Hi,

i am playing around with puppet-easticsearch 0.4.0, works wells so far
(thanks!), but I am missing a few options I havent seen in the
documentation. As I couldnt figure it out immediately by reading the
scripts, may be someone can help me fast on this:

  • there is an option to change the port (9200), but this is only the http
    port. Is there an option to change the tcp transport port as well?
  • how can I configure logging? I think about logfile names and loglevel,
    may be even thresholds for slowlog. May be this is interesting enough to
    add it to the documentation?
  • is there an option in the module to easily configure memory usage?
  • how can I configure the discovery minimum?

I am aware that I could go ahead and manipulate the elasticsearch.yml file
with puppet, I am just curious if there are options for my questions
already implemented in the module I have missed. So if someone could give
me a hint or an example it would be really helpful!

Thanks in advance!
Andrej

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/01bd5d3c-f00b-4a5d-b341-054af734462d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi Andrej,

Thank you for using the puppet module :slight_smile:

The 'port' and 'discovery minimum' settings are both configuration settings
for the elasticsearch.yml file.
You can set those in the 'config' option variable, for example:

elasticsearch::instance { 'instancename':
config => { 'http.port' => '9210', 'discovery.zen.minimum_master_nodes'
=> 3 }
}

For the logging part, management of the logging.yml file is very limited at
the moment but i hope to get some feedback on extending that.
The thresholds for the slowlogs can be set in the same config option
variable.
See Elasticsearch Platform — Find real-time answers at scale | Elastic
for more information.

If you have any further questions, let me know.

Cheers

On Thursday, June 19, 2014 9:53:10 AM UTC+1, Andrej Rosenheinrich wrote:

Hi,

i am playing around with puppet-easticsearch 0.4.0, works wells so far
(thanks!), but I am missing a few options I havent seen in the
documentation. As I couldnt figure it out immediately by reading the
scripts, may be someone can help me fast on this:

  • there is an option to change the port (9200), but this is only the http
    port. Is there an option to change the tcp transport port as well?
  • how can I configure logging? I think about logfile names and loglevel,
    may be even thresholds for slowlog. May be this is interesting enough to
    add it to the documentation?
  • is there an option in the module to easily configure memory usage?
  • how can I configure the discovery minimum?

I am aware that I could go ahead and manipulate the elasticsearch.yml file
with puppet, I am just curious if there are options for my questions
already implemented in the module I have missed. So if someone could give
me a hint or an example it would be really helpful!

Thanks in advance!
Andrej

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/41d7340c-5570-4728-b979-35f97c233e25%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Richard,

thanks for your answer, it for sure helped! Still, I am puzzling with a few
effects and questions:

1.) I am a bit confused by your class/instance idea. I can do something
pretty simple like class { 'elasticsearch' : version => '0.90.7' } and it
will install elasticsearch in the correct version using the default
settings you defined. Repeating this (I tested every step on a fresh debian
instance in a VM, no different puppet installation steps in between) with a
config added in class like

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

I still get elasticsearch installed, but it completely ignores everything
in the config. (I should be able to curl localhost:9210, but its up and
running on the old default port, using the old cluster name). You explained
overwriting for instances and classes a bit, so I tried the following thing
(again, blank image, no previous installation) :

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

elasticsearch::instance { 'es-01':
}

What happened is that I have two elasticsearch instances running, one with
the default value and another one (es-01) that uses the provided
configuration. Even freakier, I install java7 in my script before the
snippet posted , the first (default based) elasticsearch version uses the
standard openjdk-6 java, the second instance (es-01) uses java7.
So, where is my mistake or what am I doing wrong? What would be the way to
install and start only one service using provided configuration? And does
elasticsearch::instance require an instance name? I would really miss the
funny comic node names :wink:

  1. As you pointed out I can define all values from elasticsearch.yml in the
    config hash. But what about memory settings (I usually modify the init.d
    script for that), can I configure Xms and Xmx settings in the puppet module
    somehow?

Logging configuration would be a nice-to-have (no must-have), just in case
you were wondering :wink:

I hope my questions don't sound too confusing, if you could give me a hint
on what I am doing wrong I would really appreciate it.

Thanks in advance!
Andrej

Am Freitag, 20. Juni 2014 09:44:49 UTC+2 schrieb Richard Pijnenburg:

Hi Andrej,

Thank you for using the puppet module :slight_smile:

The 'port' and 'discovery minimum' settings are both configuration
settings for the elasticsearch.yml file.
You can set those in the 'config' option variable, for example:

elasticsearch::instance { 'instancename':
config => { 'http.port' => '9210', 'discovery.zen.minimum_master_nodes'
=> 3 }
}

For the logging part, management of the logging.yml file is very limited
at the moment but i hope to get some feedback on extending that.
The thresholds for the slowlogs can be set in the same config option
variable.
See
Elasticsearch Platform — Find real-time answers at scale | Elastic
for more information.

If you have any further questions, let me know.

Cheers

On Thursday, June 19, 2014 9:53:10 AM UTC+1, Andrej Rosenheinrich wrote:

Hi,

i am playing around with puppet-easticsearch 0.4.0, works wells so far
(thanks!), but I am missing a few options I havent seen in the
documentation. As I couldnt figure it out immediately by reading the
scripts, may be someone can help me fast on this:

  • there is an option to change the port (9200), but this is only the http
    port. Is there an option to change the tcp transport port as well?
  • how can I configure logging? I think about logfile names and loglevel,
    may be even thresholds for slowlog. May be this is interesting enough to
    add it to the documentation?
  • is there an option in the module to easily configure memory usage?
  • how can I configure the discovery minimum?

I am aware that I could go ahead and manipulate the elasticsearch.yml
file with puppet, I am just curious if there are options for my questions
already implemented in the module I have missed. So if someone could give
me a hint or an example it would be really helpful!

Thanks in advance!
Andrej

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dc849f26-3d9f-4f47-9a55-2b2476029b55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi Andrej,

Sorry for the late response. Didn't get an update email about it.

As long as you don't setup an instance with the 'elasticsearch::instance'
define it will only install the package but do nothing afterwards.
I recently fixed that the default files from the packages are being removed
now.
The memory can be set via the init_defaults hash by setting the ES_HEAP
option.

The issue with 0.90.x versions is that it automatically starts up after
package installation.
Since i don't stop it, it keeps running. Its advised to run a newer version
of ES since 0.90.x will be EOL'd at some point.

On Thursday, June 26, 2014 2:24:47 PM UTC+1, Andrej Rosenheinrich wrote:

Hi Richard,

thanks for your answer, it for sure helped! Still, I am puzzling with a
few effects and questions:

1.) I am a bit confused by your class/instance idea. I can do something
pretty simple like class { 'elasticsearch' : version => '0.90.7' } and it
will install elasticsearch in the correct version using the default
settings you defined. Repeating this (I tested every step on a fresh debian
instance in a VM, no different puppet installation steps in between) with a
config added in class like

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

I still get elasticsearch installed, but it completely ignores everything
in the config. (I should be able to curl localhost:9210, but its up and
running on the old default port, using the old cluster name). You explained
overwriting for instances and classes a bit, so I tried the following thing
(again, blank image, no previous installation) :

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

elasticsearch::instance { 'es-01':
}

What happened is that I have two elasticsearch instances running, one with
the default value and another one (es-01) that uses the provided
configuration. Even freakier, I install java7 in my script before the
snippet posted , the first (default based) elasticsearch version uses the
standard openjdk-6 java, the second instance (es-01) uses java7.
So, where is my mistake or what am I doing wrong? What would be the way to
install and start only one service using provided configuration? And does
elasticsearch::instance require an instance name? I would really miss the
funny comic node names :wink:

  1. As you pointed out I can define all values from elasticsearch.yml in
    the config hash. But what about memory settings (I usually modify the
    init.d script for that), can I configure Xms and Xmx settings in the puppet
    module somehow?

Logging configuration would be a nice-to-have (no must-have), just in case
you were wondering :wink:

I hope my questions don't sound too confusing, if you could give me a hint
on what I am doing wrong I would really appreciate it.

Thanks in advance!
Andrej

Am Freitag, 20. Juni 2014 09:44:49 UTC+2 schrieb Richard Pijnenburg:

Hi Andrej,

Thank you for using the puppet module :slight_smile:

The 'port' and 'discovery minimum' settings are both configuration
settings for the elasticsearch.yml file.
You can set those in the 'config' option variable, for example:

elasticsearch::instance { 'instancename':
config => { 'http.port' => '9210', 'discovery.zen.minimum_master_nodes'
=> 3 }
}

For the logging part, management of the logging.yml file is very limited
at the moment but i hope to get some feedback on extending that.
The thresholds for the slowlogs can be set in the same config option
variable.
See
Elasticsearch Platform — Find real-time answers at scale | Elastic
for more information.

If you have any further questions, let me know.

Cheers

On Thursday, June 19, 2014 9:53:10 AM UTC+1, Andrej Rosenheinrich wrote:

Hi,

i am playing around with puppet-easticsearch 0.4.0, works wells so far
(thanks!), but I am missing a few options I havent seen in the
documentation. As I couldnt figure it out immediately by reading the
scripts, may be someone can help me fast on this:

  • there is an option to change the port (9200), but this is only the
    http port. Is there an option to change the tcp transport port as well?
  • how can I configure logging? I think about logfile names and loglevel,
    may be even thresholds for slowlog. May be this is interesting enough to
    add it to the documentation?
  • is there an option in the module to easily configure memory usage?
  • how can I configure the discovery minimum?

I am aware that I could go ahead and manipulate the elasticsearch.yml
file with puppet, I am just curious if there are options for my questions
already implemented in the module I have missed. So if someone could give
me a hint or an example it would be really helpful!

Thanks in advance!
Andrej

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0b406f7a-cea1-487e-95c2-bc894b193487%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Hi Richard,

getting back to this after a while. Thanks for pointing me to the fact,
that the class itself actually does nothing than install and only instances
merge configs. This was something completely not aware to me (may be you
could add a line in the documentation?) but looking through the module I
could figure it out.
What I am wondering, what was the reason for this design decision? When I
want to install elasticsarch once on a machine I have to configure the
class (things like manage_repo can not be configured in the instance,
right?) and an instance. This will install two services, one for the class,
not caring about config, and one for the instance, what might be confusing
for some users.

Again, thanks for your help that finally got me on the right track!
Andrej

Am Dienstag, 1. Juli 2014 14:37:55 UTC+2 schrieb Richard Pijnenburg:

Hi Andrej,

Sorry for the late response. Didn't get an update email about it.

As long as you don't setup an instance with the 'elasticsearch::instance'
define it will only install the package but do nothing afterwards.
I recently fixed that the default files from the packages are being
removed now.
The memory can be set via the init_defaults hash by setting the ES_HEAP
option.

The issue with 0.90.x versions is that it automatically starts up after
package installation.
Since i don't stop it, it keeps running. Its advised to run a newer
version of ES since 0.90.x will be EOL'd at some point.

On Thursday, June 26, 2014 2:24:47 PM UTC+1, Andrej Rosenheinrich wrote:

Hi Richard,

thanks for your answer, it for sure helped! Still, I am puzzling with a
few effects and questions:

1.) I am a bit confused by your class/instance idea. I can do something
pretty simple like class { 'elasticsearch' : version => '0.90.7' } and it
will install elasticsearch in the correct version using the default
settings you defined. Repeating this (I tested every step on a fresh debian
instance in a VM, no different puppet installation steps in between) with a
config added in class like

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

I still get elasticsearch installed, but it completely ignores everything
in the config. (I should be able to curl localhost:9210, but its up and
running on the old default port, using the old cluster name). You explained
overwriting for instances and classes a bit, so I tried the following thing
(again, blank image, no previous installation) :

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

elasticsearch::instance { 'es-01':
}

What happened is that I have two elasticsearch instances running, one
with the default value and another one (es-01) that uses the provided
configuration. Even freakier, I install java7 in my script before the
snippet posted , the first (default based) elasticsearch version uses the
standard openjdk-6 java, the second instance (es-01) uses java7.
So, where is my mistake or what am I doing wrong? What would be the way
to install and start only one service using provided configuration? And
does elasticsearch::instance require an instance name? I would really miss
the funny comic node names :wink:

  1. As you pointed out I can define all values from elasticsearch.yml in
    the config hash. But what about memory settings (I usually modify the
    init.d script for that), can I configure Xms and Xmx settings in the puppet
    module somehow?

Logging configuration would be a nice-to-have (no must-have), just in
case you were wondering :wink:

I hope my questions don't sound too confusing, if you could give me a
hint on what I am doing wrong I would really appreciate it.

Thanks in advance!
Andrej

Am Freitag, 20. Juni 2014 09:44:49 UTC+2 schrieb Richard Pijnenburg:

Hi Andrej,

Thank you for using the puppet module :slight_smile:

The 'port' and 'discovery minimum' settings are both configuration
settings for the elasticsearch.yml file.
You can set those in the 'config' option variable, for example:

elasticsearch::instance { 'instancename':
config => { 'http.port' => '9210',
'discovery.zen.minimum_master_nodes' => 3 }
}

For the logging part, management of the logging.yml file is very limited
at the moment but i hope to get some feedback on extending that.
The thresholds for the slowlogs can be set in the same config option
variable.
See
Elasticsearch Platform — Find real-time answers at scale | Elastic
for more information.

If you have any further questions, let me know.

Cheers

On Thursday, June 19, 2014 9:53:10 AM UTC+1, Andrej Rosenheinrich wrote:

Hi,

i am playing around with puppet-easticsearch 0.4.0, works wells so far
(thanks!), but I am missing a few options I havent seen in the
documentation. As I couldnt figure it out immediately by reading the
scripts, may be someone can help me fast on this:

  • there is an option to change the port (9200), but this is only the
    http port. Is there an option to change the tcp transport port as well?
  • how can I configure logging? I think about logfile names and
    loglevel, may be even thresholds for slowlog. May be this is interesting
    enough to add it to the documentation?
  • is there an option in the module to easily configure memory usage?
  • how can I configure the discovery minimum?

I am aware that I could go ahead and manipulate the elasticsearch.yml
file with puppet, I am just curious if there are options for my questions
already implemented in the module I have missed. So if someone could give
me a hint or an example it would be really helpful!

Thanks in advance!
Andrej

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2076ad10-ba54-4cb2-9912-579202b48dae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Hi Richard,

another question: you are creating the elasticsearch user and group
somewhere in the module (havent found exactly where yet). My problem is
that I have to create a directory for data_dir (on a different device) that
is needed by the class (or instance, not sure), but I need the owner and
the group to be able to set it otherwise the service won't start. Can I set
a requirement in my file declaration to make sure that the user and the
group already exist? Something like

file { "/data/elasticsearch":
ensure => "directory",
owner => "elasticsearch",
group => "elasticsearch",
require => ???
}

Once again, thanks!
Andrej

Am Dienstag, 1. Juli 2014 14:37:55 UTC+2 schrieb Richard Pijnenburg:

Hi Andrej,

Sorry for the late response. Didn't get an update email about it.

As long as you don't setup an instance with the 'elasticsearch::instance'
define it will only install the package but do nothing afterwards.
I recently fixed that the default files from the packages are being
removed now.
The memory can be set via the init_defaults hash by setting the ES_HEAP
option.

The issue with 0.90.x versions is that it automatically starts up after
package installation.
Since i don't stop it, it keeps running. Its advised to run a newer
version of ES since 0.90.x will be EOL'd at some point.

On Thursday, June 26, 2014 2:24:47 PM UTC+1, Andrej Rosenheinrich wrote:

Hi Richard,

thanks for your answer, it for sure helped! Still, I am puzzling with a
few effects and questions:

1.) I am a bit confused by your class/instance idea. I can do something
pretty simple like class { 'elasticsearch' : version => '0.90.7' } and it
will install elasticsearch in the correct version using the default
settings you defined. Repeating this (I tested every step on a fresh debian
instance in a VM, no different puppet installation steps in between) with a
config added in class like

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

I still get elasticsearch installed, but it completely ignores everything
in the config. (I should be able to curl localhost:9210, but its up and
running on the old default port, using the old cluster name). You explained
overwriting for instances and classes a bit, so I tried the following thing
(again, blank image, no previous installation) :

class { 'elasticsearch' :
version => '0.90.7',
config => {
'cluster' => {
'name' => 'andrejtest'
},
'http.port' => '9210'
}
}

elasticsearch::instance { 'es-01':
}

What happened is that I have two elasticsearch instances running, one
with the default value and another one (es-01) that uses the provided
configuration. Even freakier, I install java7 in my script before the
snippet posted , the first (default based) elasticsearch version uses the
standard openjdk-6 java, the second instance (es-01) uses java7.
So, where is my mistake or what am I doing wrong? What would be the way
to install and start only one service using provided configuration? And
does elasticsearch::instance require an instance name? I would really miss
the funny comic node names :wink:

  1. As you pointed out I can define all values from elasticsearch.yml in
    the config hash. But what about memory settings (I usually modify the
    init.d script for that), can I configure Xms and Xmx settings in the puppet
    module somehow?

Logging configuration would be a nice-to-have (no must-have), just in
case you were wondering :wink:

I hope my questions don't sound too confusing, if you could give me a
hint on what I am doing wrong I would really appreciate it.

Thanks in advance!
Andrej

Am Freitag, 20. Juni 2014 09:44:49 UTC+2 schrieb Richard Pijnenburg:

Hi Andrej,

Thank you for using the puppet module :slight_smile:

The 'port' and 'discovery minimum' settings are both configuration
settings for the elasticsearch.yml file.
You can set those in the 'config' option variable, for example:

elasticsearch::instance { 'instancename':
config => { 'http.port' => '9210',
'discovery.zen.minimum_master_nodes' => 3 }
}

For the logging part, management of the logging.yml file is very limited
at the moment but i hope to get some feedback on extending that.
The thresholds for the slowlogs can be set in the same config option
variable.
See
Elasticsearch Platform — Find real-time answers at scale | Elastic
for more information.

If you have any further questions, let me know.

Cheers

On Thursday, June 19, 2014 9:53:10 AM UTC+1, Andrej Rosenheinrich wrote:

Hi,

i am playing around with puppet-easticsearch 0.4.0, works wells so far
(thanks!), but I am missing a few options I havent seen in the
documentation. As I couldnt figure it out immediately by reading the
scripts, may be someone can help me fast on this:

  • there is an option to change the port (9200), but this is only the
    http port. Is there an option to change the tcp transport port as well?
  • how can I configure logging? I think about logfile names and
    loglevel, may be even thresholds for slowlog. May be this is interesting
    enough to add it to the documentation?
  • is there an option in the module to easily configure memory usage?
  • how can I configure the discovery minimum?

I am aware that I could go ahead and manipulate the elasticsearch.yml
file with puppet, I am just curious if there are options for my questions
already implemented in the module I have missed. So if someone could give
me a hint or an example it would be really helpful!

Thanks in advance!
Andrej

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30117937-d76f-41fb-a8e3-a01f06d0e1f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Hello Andrej,

I am trying to install multinode cluster using puppet module, I don't seems its working, because It does not give you option to assign different node names to different instances also, seems like module is written in a way to have multiple instances on single VM or server.