[SOLVED] Mail server configuration issues

alerting

#1

Greetings,

I followed the guides to install the ELK stack and everything went according to expectations. I currently have 2 kibana4 nodes with elasticsearch 1.7.1 query nodes, 2 elasticsearch 1.7.1 data nodes and 2 logstash 1.5.4 nodes. Everything is working brilliantly. Cluster health shows all green.

Next step is installing and configuring watcher. I installed the latest license and latest watcher.
Curl output to show watcher functioning:

> curl -XGET 'http://localhost:9200/_watcher/stats?pretty=true'
> {
>   "watcher_state" : "started",
>   "watch_count" : 1,
>   "execution_thread_pool" : {
>     "queue_size" : 0,
>     "max_size" : 10
>   }
> }

The watch that was curled in:

> curl -XPUT 'http://localhost:9200/_watcher/watch/cluster_health_watch' -d '{
>   "trigger" : {
>     "schedule" : { "interval" : "10s" }
>   },
>   "input" : {
>     "http" : {
>       "request" : {
>        "host" : "localhost",
>        "port" : 9200,
>        "path" : "/_cluster/health"
>       }
>     }
>   },
>   "condition" : {
>     "compare" : {
>       "ctx.payload.status" : { "eq" : "red" }
>     }
>   },
>   "actions" : {
>     "send_email" : {
>       "email" : {
>         "to" : "<receivingaddress>@gmail.com",
>         "subject" : "Cluster Status Warning",
>         "body" : "Cluster status is RED"
>       }
>     }
>   }
> }'

So far so good! However when adding below mailserver configuration to the elasticsearch.yml of every node, elasticsearch refuses to start and no log messages are added to elasticsearch's logfles.

> watcher.actions.email.service.account:
>     gmail_account:
>         profile: gmail
>         smtp:
>             auth: true
>             starttls.enable: true
>             host: smtp.gmail.com
>             port: 587
>       user: <username@gmail.com>
>       password: <password>

Gmail's allow less secure apps is turned on.
Taking out the above mailserver config from elasticsearch.yml lets Elasticsearch start normally.

Can anyone more knowledgable point me to what I'm doing wrong?


(Mark Walkom) #2

How are you starting ES?

Actually I think the issue is the white spacing of the user and password fields, they need to be aligned with port.


#3

Service elasticsearch start or /etc/init.d/elasticsearch start. Both return a status of not running after executing.
It is my understanding that starting ES with bin/elasticsearch start, is ill advised and throws me log4j errors(and nothing mentioning anything else) when I try.

I tried your suggestion of aligning the user and password fields with port (which was excellent btw and I'm hitting myself for overlooking that) as below:

> watcher.actions.email.service.account:
>     gmail_account:
>         profile: gmail
>         smtp:
>             auth: true
>             starttls.enable: true
>             host: smtp.gmail.com
>             port: 587
>             user: <sendingaddress>@gmail.com	 
>             password: <password>

But alas the issue persists.
I'm starting to think I'm operating on the wrong assumption that I can just add the mail configuration at the bottom of the elasticsearch.yml.

As an append. Elasticsearch stops normally before making changes as shown in log below:

> [2015-09-10 13:55:33,413][INFO ][node                     ] [esclient-2] stopping ...
> [2015-09-10 13:55:33,483][INFO ][node                     ] [esclient-2] stopped
> [2015-09-10 13:55:33,484][INFO ][node                     ] [esclient-2] closing ...
> [2015-09-10 13:55:33,492][INFO ][node                     ] [esclient-2] closed

No logs were added when trying to start ES after adding the mail config at 2015-09-10 14:10:00.


(Mark Walkom) #4

When I have run into these I try to run the ES command directly myself, it sometimes spits out a little more.

eg something like this, but not this as it's running on my laptop :wink:
/usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/Users/markw/Workspace/elastic/elasticsearch-1.7.1 -cp :/Users/markw/Workspace/elastic/elasticsearch-1.7.1/lib/elasticsearch-1.7.1.jar:/Users/markw/Workspace/elastic/elasticsearch-1.7.1/lib/*:/Users/markw/Workspace/elastic/elasticsearch-1.7.1/lib/sigar/* -Des.config=Confs/ES/elasticsearch.yml -Xmx2g -Xms2g org.elasticsearch.bootstrap.Elasticsearch


#5

As suggested, started ES directly.

> sudo bin/elasticsearch start
> Failed to configure logging...
> org.elasticsearch.ElasticsearchException: Failed to load logging configuration
>     at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:142)
>     at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:92)
>     at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:131)
>     at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:216)
>     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
> Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
>     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>     at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
>     at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
>     at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>     at java.nio.file.Files.readAttributes(Files.java:1737)
>     at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
>     at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
>     at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
>     at java.nio.file.Files.walkFileTree(Files.java:2662)
>     at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:126)
>     ... 4 more
> log4j:WARN No appenders could be found for logger (node).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

#6

Update:
For any future users running into the same issue. Make sure to check your index creation for watcher as show in http://hadooptutorials.co.in/tutorials/elasticsearch/real-time-alerting-using-elasticsearch-watcher.html
completes successfully. This was causing the above problem even though there were no logs to indicate as such.


(Mark Walkom) #7

Which index specifically are you referring to?


#8

Under the paragraph "Creating an index" in the linked tutorial. http://localhost:9200/event. Strangely enough this allowed ES to start with the mail configuration.

Issue is solved. On to new different issues.

Thanks for the help Warkolm!


(system) #9