Error during elasticsearch deployment via puppet

shoshoro99 · November 16, 2017, 7:50pm

Deploying elasticsearch via puppet ( puppet version 4.10.8) elasticsearch module (elastic-elasticsearch (v5.4.3)) , i got te below errors

Info: Applying configuration version '1510860503'
Error: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Augeas[defaults_elastic1]: Could not evaluate: Saving failed, see debug
Notice: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Exec[systemd_reload_elastic1]: Dependency Augeas[defaults_elastic1] has failures: true
Warning: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Exec[systemd_reload_elastic1]: Skipping because of failed dependencies
Notice: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Service[elasticsearch-instance-elastic1]: Dependency Augeas[defaults_elastic1] has failures: true
Warning: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Service[elasticsearch-instance-elastic1]: Skipping because of failed dependencies
Notice: Applied catalog in 1.01 seconds

when i ran debug, i got the below messages

Debug: Augeas[defaults_elastic1](provider=augeas): Put failed on one or more files, output from /augeas//error:
Debug: Augeas[defaults_elastic1](provider=augeas): /augeas/files/etc/sysconfig/elasticsearch-elastic1/error = put_failed
Debug: Augeas[defaults_elastic1](provider=augeas): /augeas/files/etc/sysconfig/elasticsearch-elastic1/error/path = /files/etc/sysconfig/elasticsearch-elastic1
Debug: Augeas[defaults_elastic1](provider=augeas): /augeas/files/etc/sysconfig/elasticsearch-elastic1/error/lens = /opt/puppetlabs/puppet/share/augeas/lenses/dist/shellvars.aug:194.12-.60:
Debug: Augeas[defaults_elastic1](provider=augeas): /augeas/files/etc/sysconfig/elasticsearch-elastic1/error/message = Failed to match

<<rec:|       { /@if/ = /([^\001-\004\t\n "#';]|[^\001-\004\t\n "#';]([^\001-\004\n"#';]|("([^\001-\004"\\]|\\\\[^\001-\004\n]|[\t ]*\\\\\n[\t ]*)*"|'[^\001-\004']*')+|`[^\001-\004\n`]*`|``[^\001-\004\t\n ;`]+``|\\$\\([^\001-\004\n#\\]*\\)|\\$\\(\\([^\001-\004\n#)\\]*\\)\\))+[^\001-\004\t\n "#';])/ }
        | { /@for/ = /([^\001-\004\t\n "#';]|[^\001-\004\t\n "#';]([^\001-\004\n"#';]|("([^\001-\004"\\]|\\\\[^\001-\004\n]|[\t ]*\\\\\n[\t ]*)*"|'[^\001-\004']*')+|`[^\001-\004\n`]*`|``[^\001-\004\t\n ;`]+``|\\$\\([^\001-\004\n#\\]*\\)|\\$\\(\\([^\001-\004\n#)\\]*\\)\\))+[^\001-\004\t\n "#';])/ }
        | { /@select/ = /([^\001-\004\t\n "#';]|[^\001-\004\t\n "#';]([^\001-\004\n"#';]|("([^\001-\004"\\]|\\\\[^\001-\004\n]|[\t ]*\\\\\n[\t ]*)*"|'[^\001-\004']*')+|`[^\001-\004\n`]*`|``[^\001-\004\t\n ;`]+``|\\$\\([^\001-\004\n#\\]*\\)|\\$\\(\\([^\001-\004\n#)\\]*\\)\\))+[^\001-\004\t\n "#';])/ }
        | { /@while/ = /([^\001-\004\t\n "#';]|[^\001-\004\t\n "#';]([^\001-\004\n"#';]|("([^\001-\004"\\]|\\\\[^\001-\004\n]|[\t ]*\\\\\n[\t ]*)*"|'[^\001-\004']*')+|`[^\001-\004\n`]*`|``[^\001-\004\t\n ;`]+``|\\$\\([^\001-\004\n#\\]*\\)|\\$\\(\\([^\001-\004\n#)\\]*\\)\\))+[^\001-\004\t\n "#';])/ }
        | { /@until/ = /([^\001-\004\t\n "#';]|[^\001-\004\t\n "#';]([^\001-\004\n"#';]|("([^\001-\004"\\]|\\\\[^\001-\004\n]|[\t ]*\\\\\n[\t ]*)*"|'[^\001-\004']*')+|`[^\001-\004\n`]*`|``[^\001-\004\t\n ;`]+``|\\$\\([^\001-\004\n#\\]*\\)|\\$\\(\\([^\001-\004\n#)\\]*\\)\\))+[^\001-\004\t\n "#';])/ }
        | { /@case/ = /(([^\001-\004\t\n "'-);`]|\\\\")+|"([^\001-\004\t\n "'-);`]|\\\\")+")/ }
        | { /@function/ = /[.0-9A-Z_a-z-]+/ }>>)*
  with tree
    { "CONF_DIR" = "/etc/elasticsearch/elastic1" } { "DATA_DIR" = "/var/lib/elasticsearch" } { "ES_GROUP" = "elasticsearch" } { "ES_HEAP_SIZE" = "1g" } { "ES_HOME" = "/usr/share/elasticsearch" } { "ES_USER" = "elasticsearch" } { "LOG_DIR" = "/var/log/elasticsearch/elastic1" } { "MAX_LOCKED_MEMORY" = "unlimited" } { "MAX_MAP_COUNT" = "262144" } { "MAX_OPEN_FILES" = "65535" } { "ES_JVM_OPTIONS" = "/etc/elasticsearch/elastic1/jvm.options" } { "ES_PATH_CONF" = "/etc/elasticsearch/elastic1" } { "MAX_THREADS" = "4096" } { "jvm_options" = "[ "-Xms1g, -Xmx1g"]" }
Debug: Augeas[defaults_elastic1](provider=augeas): Closed the augeas connection
Error: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Augeas[defaults_elastic1]: Could not evaluate: Saving failed, see debug
Debug: Prefetching ruby resources for elasticsearch_service_file
Notice: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Exec[systemd_reload_elastic1]: Dependency Augeas[defaults_elastic1] has failures: true
Warning: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Exec[systemd_reload_elastic1]: Skipping because of failed dependencies
Debug: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Exec[systemd_reload_elastic1]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Service[elasticsearch-instance-elastic1]: Dependency Augeas[defaults_elastic1] has failures: true
Warning: /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Service[elasticsearch-instance-elastic1]: Skipping because of failed dependencies

please can anyone put me in the right direction on how to resolve this.
i am just a beginner

tylerjl · November 17, 2017, 4:06pm

Hi @shoshoro99, this is likely due to the way you're passing in jvm_options to your init defaults. Rather than defining jvm_options in that way, use the supported jvm_options parameter to control those settings you're trying to control. Does that change anything?

PS: I edited your post to wrap Puppet's output in code blocks, considering doing so yourself in the future as it makes reading code output much easier.

shoshoro99 · December 1, 2017, 10:17pm

thanks for our reply, i did apply your suggestion, but got different error listed below

    Info:        /Stage[main]/Analytic::Elasticsearch/Elasticsearch::Instance[elastic1]/Elasticsearch::Service[elastic1]/Elasticsearch::Service::Systemd[elastic1]/Service[elasticsearch-instance-elastic1]: Unscheduling refresh on Service[elasticsearch-instance-elastic1]
            Notice: Failed to connect to the ES instance within timeout window of 10 seconds; giving up.
            Error: Failed to apply catalog: Connection refused - connect(2) for "localhost" port 9200

below is my puppet manifest

    class analytic::elasticsearch {
    include ::java
    class { 'elasticsearch':

      manage_repo  => true,
      repo_version => '6.x',
    }
    elasticsearch::instance { $hostname:
      config        => {
               'cluster.name' => 'analytics',
               'node.name' => $fqdn,
               'network.host' => ['127.0.0.1' ],
               'bootstrap.memory_lock' => true ,
              }, # Configuration hash
      jvm_options =>
           [
        '-Xms1g',
        '-Xmx1g'
          ]
          
      datadir       => ['/home/data' ] # Data directory
                  }
    elasticsearch::index { $hostname:
               settings   => {
                  'index' => {
                            'number_of_shards'=> '1',
                            'number_of_replicas' => '0',
                            'refresh_interval' => '5',
                            'codec' => 'best_compression'
                            }
                            }
                        }
    }

tylerjl · December 4, 2017, 5:00pm

@shoshoro99 a timeout would mean that in attempting to manage your index settings, Puppet could not reach Elasticsearch in time to apply the settings for the index. If you run that manifest a second time, does the catalog apply cleanly? If so, that would mean that Elasticsearch simply took a long time to come up, and you can deal with that in the future by using the timeout parameter on elasticsearch::index resources.

shoshoro99 · December 29, 2017, 4:36am

thanks for your reply tylerj, i tried API_timeout increase from 10 to 50, i still get same error could not connect to local host on 9200 error, also what i have observed is that i see two services in this path /usr/lib/systemd/system/elasticsearch.services and elasticsearch-elastic1.service ,the latter is instance based services (where elastic1 is my instance name) i am confused. pardon me i am just a begginer with elasticsearch and puppet module elastic-elasticsearch --version 6.1.0

tylerjl · January 2, 2018, 10:20pm

@shoshoro99 you are correct, the Puppet module manages separate instances for Elasticsearch, so elasticsearch.service will be unused on the target system, and Puppet will manage and start elasticsearch-elastic1 instead.

Regarding the timeout issue, is the Elasticsearch API up and available on the system after Puppet runs? That is, does a simple curl localhost:9200 get a response? You may also want to check if it's listening with something like netstat -lpnt | grep 9200.

shoshoro99 · January 3, 2018, 3:44am

thanks, but when i start elasticsearch-elsatic1.service, i got below error and there is this specific line in bold <asticsearch[5059]: **Error: Could not find or load main class 2g>.

elasticsearch-elastic1.service - Elasticsearch instance elastic1
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch-elastic1.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2018-01-02 21:10:40 CST; 2s ago
     Docs: http://www.elastic.co
  Process: 5059 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch-elastic1.pid --quiet (code=exited, status=1/FAILURE)
  Process: 5057 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=203/EXEC)
 Main PID: 5059 (code=exited, status=1/FAILURE)

Jan 02 21:10:40 elastic1.shoshoro99.com systemd[1]: Starting Elasticsearch instance elastic1...
Jan 02 21:10:40 elastic1.shoshoro99.com systemd[5057]: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-s...ectory
Jan 02 21:10:40 elastic1.shoshoro99.com systemd[1]: Started Elasticsearch instance elastic1.
Jan 02 21:10:40 elastic1.shoshoro99.com elasticsearch[5059]: **Error: Could not find or load main class 2g**
Jan 02 21:10:40 elastic1.shoshoro99.com systemd[1]: elasticsearch-elastic1.service: main process exited, code=exited, status=1/FAILURE
Jan 02 21:10:40 elastic1.shoshoro99.com systemd[1]: Unit elasticsearch-elastic1.service entered failed state.
Jan 02 21:10:40 elastic1.shoshoro99.com systemd[1]: elasticsearch-elastic1.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

And when i check curl http://localhost:9200, it return below error
curl: (7) Failed connect to localhost:9200; Connection refused
please advice what next should i check.

tylerjl · January 3, 2018, 4:35pm

Error: Could not find or load main class 2g is something I haven't seen before. What version of Java are you running on this machine?

$ java -version

shoshoro99 · January 3, 2018, 5:02pm

here is my java version installed
[root@elastic1 ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
also i used puppet to deploy this instance
puppet master and agent version
4.10.9

tylerjl · January 3, 2018, 5:57pm

It's definitely a strange error, I'm not sure where it may be originating from. I'd perhaps also check Elasticsearch's logs (which are in /var/log/elasticsearch by default) and also check the contents of /etc/sysconfig/elasticsearch-elastic1.

shoshoro99 · January 3, 2018, 6:28pm

here is content of /etc/sysconfig/elasticsearch-elastic1

        t@elastic1 ~]# cat /etc/sysconfig/elasticsearch-elastic1
        CONF_DIR=/etc/elasticsearch/elastic1
        DATA_DIR=/var/lib/elasticsearch
        ES_GROUP=elasticsearch
        ES_HOME=/usr/share/elasticsearch
        ES_JAVA_OPTS=2g
        ES_JVM_OPTIONS=/etc/elasticsearch/elastic1/jvm.options
        ES_PATH_CONF=/etc/elasticsearch/elastic1
        ES_USER=elasticsearch
        LOG_DIR=/var/log/elasticsearch/elastic1
        MAX_LOCKED_MEMORY=unlimited
        MAX_MAP_COUNT=262144
        MAX_OPEN_FILES=65535
        MAX_THREADS=4096

and /var/log/elasticsearch/elastic1/*

    @elastic1 ~]# cat /var/log/elasticsearch/elastic1/*
    OpenJDK 64-Bit Server VM (25.151-b12) for linux-amd64 JRE (1.8.0_151-b12), built on Dec 19 2017 17:16:52 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-16)
    Memory: 4k page, physical 3881808k(3198392k free), swap 3145724k(3145724k free)
    CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=348966912 -XX:MaxTenuringThreshold=6 -XX:NewSize=348966912 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=697933824 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
    Heap
     par new generation   total 306688K, used 10905K [0x0000000080000000, 0x0000000094cc0000, 0x0000000094cc0000)
      eden space 272640K,   4% used [0x0000000080000000, 0x0000000080aa6688, 0x0000000090a40000)
      from space 34048K,   0% used [0x0000000090a40000, 0x0000000090a40000, 0x0000000092b80000)
      to   space 34048K,   0% used [0x0000000092b80000, 0x0000000092b80000, 0x0000000094cc0000)
     concurrent mark-sweep generation total 1756416K, used 0K [0x0000000094cc0000, 0x0000000100000000, 0x0000000100000000)
     Metaspace       used 2756K, capacity 4480K, committed 4480K, reserved 1056768K
      class space    used 273K, capacity 384K, committed 384K, reserved 1048576K

there is nothing stands out starangely

tylerjl · January 3, 2018, 6:41pm

The ES_JAVA_OPTS line in /etc/sysconfig/elasticsearch-elastic1 is the problem. If you remove that, Elasticsearch should start successfully.

shoshoro99 · January 3, 2018, 7:18pm

thanks it running , but when puppet agent runs again , it goes back to start given errors , as i have shown it before in my manifest i did not set ES_JAVA_OPTS, i believe somehow in the elasticsearch puppet module has some configuration that is setting this options. would kindly take a look.
below is my elasticsearch.pp

class { 'elasticsearch':
  manage_repo  => true,
  repo_version => '6.x',
  version  => '6.1.1',

}
elasticsearch::instance { $hostname:
  status  => enabled,
  config        => {
          'cluster.name' => 'analytics',
           'node.name' => $fqdn,
           'node.master' => true,
           'node.data' => false,
           'network.host' => [ $::ipaddress ],
           'network.bind_host'=> "0",
           'http.port' => '9200-9300',
           'bootstrap.memory_lock' => true ,
           'discovery.zen.ping.multicast.enabled' => false,
           'discovery.zen.ping.unicast.hosts' => ["10.0.0.64", "10.0.0.65"], # Configuration hash
           'discovery.zen.minimum_master_nodes' => '1',
                   },
  jvm_options => [
                  '-Xms2g',
                  '-Xmx2g'
                 ],
  init_defaults => {

        'MAX_OPEN_FILES' => 65535,
        'MAX_LOCKED_MEMORY' => 'unlimited',
        'MAX_MAP_COUNT' => 262144,
                 }, # Init defaults hash

}



}

tylerjl · January 3, 2018, 9:32pm

The module doesn't manage ES_JAVA_OPTS in the init defaults file; the only values it will manage are those defined in your init_defaults parameter. Perhaps Elasticsearch is failing to start due to a different error?

system · January 31, 2018, 9:32pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.