[SOLVED] Mail server configuration issues

Elastic Stack Elasticsearch
1

Greetings,

I followed the guides to install the ELK stack and everything went according to expectations. I currently have 2 kibana4 nodes with elasticsearch 1.7.1 query nodes, 2 elasticsearch 1.7.1 data nodes and 2 logstash 1.5.4 nodes. Everything is working brilliantly. Cluster health shows all green.

Next step is installing and configuring watcher. I installed the latest license and latest watcher.
Curl output to show watcher functioning:

> curl -XGET 'http://localhost:9200/_watcher/stats?pretty=true'
> {
>   "watcher_state" : "started",
>   "watch_count" : 1,
>   "execution_thread_pool" : {
>     "queue_size" : 0,
>     "max_size" : 10
>   }
> }

The watch that was curled in:

> curl -XPUT 'http://localhost:9200/_watcher/watch/cluster_health_watch' -d '{
>   "trigger" : {
>     "schedule" : { "interval" : "10s" }
>   },
>   "input" : {
>     "http" : {
>       "request" : {
>        "host" : "localhost",
>        "port" : 9200,
>        "path" : "/_cluster/health"
>       }
>     }
>   },
>   "condition" : {
>     "compare" : {
>       "ctx.payload.status" : { "eq" : "red" }
>     }
>   },
>   "actions" : {
>     "send_email" : {
>       "email" : {
>         "to" : "<receivingaddress>@gmail.com",
>         "subject" : "Cluster Status Warning",
>         "body" : "Cluster status is RED"
>       }
>     }
>   }
> }'

So far so good! However when adding below mailserver configuration to the elasticsearch.yml of every node, elasticsearch refuses to start and no log messages are added to elasticsearch's logfles.

> watcher.actions.email.service.account:
>     gmail_account:
>         profile: gmail
>         smtp:
>             auth: true
>             starttls.enable: true
>             host: smtp.gmail.com
>             port: 587
>       user: <username@gmail.com>
>       password: <password>

Gmail's allow less secure apps is turned on.
Taking out the above mailserver config from elasticsearch.yml lets Elasticsearch start normally.

Can anyone more knowledgable point me to what I'm doing wrong?

2

How are you starting ES?

Actually I think the issue is the white spacing of the user and password fields, they need to be aligned with port.

3

Service elasticsearch start or /etc/init.d/elasticsearch start. Both return a status of not running after executing.
It is my understanding that starting ES with bin/elasticsearch start, is ill advised and throws me log4j errors(and nothing mentioning anything else) when I try.

I tried your suggestion of aligning the user and password fields with port (which was excellent btw and I'm hitting myself for overlooking that) as below:

> watcher.actions.email.service.account:
>     gmail_account:
>         profile: gmail
>         smtp:
>             auth: true
>             starttls.enable: true
>             host: smtp.gmail.com
>             port: 587
>             user: <sendingaddress>@gmail.com	 
>             password: <password>

But alas the issue persists.
I'm starting to think I'm operating on the wrong assumption that I can just add the mail configuration at the bottom of the elasticsearch.yml.

As an append. Elasticsearch stops normally before making changes as shown in log below:

> [2015-09-10 13:55:33,413][INFO ][node                     ] [esclient-2] stopping ...
> [2015-09-10 13:55:33,483][INFO ][node                     ] [esclient-2] stopped
> [2015-09-10 13:55:33,484][INFO ][node                     ] [esclient-2] closing ...
> [2015-09-10 13:55:33,492][INFO ][node                     ] [esclient-2] closed

No logs were added when trying to start ES after adding the mail config at 2015-09-10 14:10:00.

4

When I have run into these I try to run the ES command directly myself, it sometimes spits out a little more.

eg something like this, but not this as it's running on my laptop :wink:
/usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/Users/markw/Workspace/elastic/elasticsearch-1.7.1 -cp :/Users/markw/Workspace/elastic/elasticsearch-1.7.1/lib/elasticsearch-1.7.1.jar:/Users/markw/Workspace/elastic/elasticsearch-1.7.1/lib/*:/Users/markw/Workspace/elastic/elasticsearch-1.7.1/lib/sigar/* -Des.config=Confs/ES/elasticsearch.yml -Xmx2g -Xms2g org.elasticsearch.bootstrap.Elasticsearch

5

As suggested, started ES directly.

> sudo bin/elasticsearch start
> Failed to configure logging...
> org.elasticsearch.ElasticsearchException: Failed to load logging configuration
>     at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:142)
>     at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:92)
>     at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:131)
>     at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:216)
>     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
> Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
>     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>     at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
>     at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
>     at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>     at java.nio.file.Files.readAttributes(Files.java:1737)
>     at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
>     at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
>     at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
>     at java.nio.file.Files.walkFileTree(Files.java:2662)
>     at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:126)
>     ... 4 more
> log4j:WARN No appenders could be found for logger (node).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
6

Update:
For any future users running into the same issue. Make sure to check your index creation for watcher as show in http://hadooptutorials.co.in/tutorials/elasticsearch/real-time-alerting-using-elasticsearch-watcher.html
completes successfully. This was causing the above problem even though there were no logs to indicate as such.

7

Which index specifically are you referring to?

8

Under the paragraph "Creating an index" in the linked tutorial. http://localhost:9200/event. Strangely enough this allowed ES to start with the mail configuration.

Issue is solved. On to new different issues.

Thanks for the help Warkolm!