Watcher in x-pack returns NullPointerException

Elastic Stack Elasticsearch
1

Hi!

I would like to ask question related to watcher in x-pack.

My environment is

x-pack -5.0.0 alpha5
elasticsearch 5.0.0alpha5

I get below error when watcher is executed.

[2016-09-12 16:43:12,322][ERROR][xpack.watcher.actions.email] [v1_l7Fa] failed to execute action [log_errors2/email_administrator]
java.lang.NullPointerException
	at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111)
	at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:70)
	at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:64)
	at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85)
	at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:109)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:404)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:280)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:476)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:450)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Watcher API(execute) shows that condition : status : success but email is not sent to root@localhost.localadmin. This is the same server as where elasticsearch is installed.

I have checked that below command is success and mail was sent successfully. So my mailserver postfix is working.

echo "test" | mail root@localhost.localadmin

What is this NullPointException is exactly referring to?

2

I was wondering , what is below line doing??

at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111)

Does anybody know?

3

Hey,

can you share your configuration as well as the email sending part of your watch? Does this still happen with rc1?

--Alex

4

Yes it does.

My sending part is below. Also this is a environment where proxy exists.

"actions" : {
  "email_administrator" : {
    "from" : "test@localhost.localdomain",
   "to" : "y-watanabe@localhost.localdomain",
    "subject" : "Encountered errors",
   "body" : "Too many error in the system, see attached data",
   "attachments" : {
     "attached_data": {
       "data" : {"format" : "json"}
     },
     "priority" : "high"
  }
}

Is there any way to change the log level of watcher?
I need to know which parameter is the root cause.

5

Can you also share your elasticsearch.yml configuration for xpack?

6

@spinscale

Sorry for the late reply.

Below is the elasticsearch.yml
I changed the ip information due to security reasons.

 [root@10028 elasticsearch]# cat elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
# インデックス領域を定義する
path.data: /data/elasticsearch/data
#
# Path to log files:
#
# ログ領域を定義する
#path.logs: /path/to/logs
path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
#node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

# General XPack setting
xpack.http.proxy.host: 192.168.1.1
xpack.http.proxy.port: 8080

# XPack watcher setting
xpack.watcher.enabled: false
xpack.notification.email.account:
  work:
    profile: standard
    smtp:
      host: 192.168.1.34
      port: 25
7

Below is the latest error log.

[2016-11-09T12:51:39,988][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [I2nQ1Xs] failed to execute action [fw/email_administrator]
java.lang.NullPointerException
        at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:68) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:62) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:161) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:63) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
8

Hi,

As mentioned above. I've also having java.lang.NullPointerExceptions with latest X-pack release 5.0.0 on CentOS7. Also my postfix config is working and currently no firewall is running.

@YuWatanabe @spinscale Did you already found a solution for this ?

Failed to execute action [cluster_health_watch/send_mail]
java.lang.NullPointerException
at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:68) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:62) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:161) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:496) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444) [elasticsearch-5.0.0.jar:5.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

9

After upgrading the Elastic stack (elasticsearch, kibana and x-pack) to level 5.0.1 mail functionality is working as a charm :slight_smile:

10

Hey Yu,

this cannot be your configuration, given that watcher is disabled? Or is that the only difference?

In addition your actions snippet is also not valid for watcher. Can you please share the whole watch?

With fixing your configuration and adding a valid watch I was again able to send an email.

You can increase logging by configuring config/log4j2.properties and setting

logger.action.name = org.elasticsearch.xpack.watcher
logger.action.level = debug

@arnold79 are you able to share your configuration and your watch as well so I can try to reproduce with 5.0.0 and ensure everything is fixed as it should be?

--Alex

11

My e-mail configuration was working after implementing 5.0.1 for elasticsearch, kibana and x-pack components.

Here is my configuration (sample code) . Hopefully this helps

Watcher trigger/input/condition/actions

PUT _xpack/watcher/watch/cluster_health_watch
{
  "trigger": {
    "schedule": { "interval": "10s" }
  },
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9200,
        "path": "/_cluster/health",
        "auth": {
          "basic": {
            "username": "elastic",
            "password": "updateme"
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.status": { "eq":"yellow"}
    }
  },
  "actions": {
    "send_mail": {
      "email": {
        "to": "username@gmail.com",
        "subject": "Elastic Stack Alert",
        "body": "Cluster status is Yellow"
      }
    }
  }
}


GET .watcher-history*/_search
{
  "sort": [
    { "result.execution_time": "desc" }
    ]
}

DELETE _xpack/watcher/watch/cluster_health_watch

elasticsearch.yml configuration

xpack.notification.email.account:
  local_account:
    email_defaults:
      from: xpack@server2.nl
    profile: standard
    smtp:
      host: 127.0.0.1
      port: 25

e-mail configuration on CentOS7

Default postfix and mail relyhost configured only !!!

12

Below is the watcher definition. Nothing special really.

PUT _xpack/watcher/watch/fw
{
  "metadata": {
    "color" : "red"
  },
  "trigger" : {
    "schedule" : {
      "interval" : "10m" 
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : "poc-fw-gaibu-ext",
        "body" : {
          "query" : {
            "bool" : {
              "must" : [
                 {"range" : {"@timestamp" : {"gte" : "now-31d", "lte" : "now"}}}
              ]
            }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : {"ctx.payload.hits.total" : {"gt" : 0 } }
  },
  "actions" : {
    "email_administrator" : {
      "email" : {
        "from": "log.analyze.kensa.bsj@bridgestone.com",
        "to"  : "y-watanabe@creationline.com",
        "subject" : "Encountered errors",
        "body" : "Too many error in the system, see attached data",
        "attachments": {	
          "attached_data" : {
            "data" : {"format" : "json" }
          }
        },
        "priority" : "high"
      }
    }
  }
}
13

@arnold79 @spinscale

My problem is occurring at customer's POC site. I do not have a access but I will check error still occurs after upgrade to 5.0.1.

If still happens , I will check the debug log.

14

@arnold79

Do you have any proxy setting enabled in your elasticsearch server ?
I know smtp is not related to proxy but would like to know any other major difference other than firewall.

15

i.e. my elasticsearch.yml is default and only updated with the x-pack additions (in this case).

No proxy enabled,but I has NAT based internet access (VirtualBox default) and yes the local firewall of CENTOS7 is running with a exclude for smtp service.

 firewall-cmd --permanent --add-service=smtp
firewall-cmd --reload

Next Wednesday I have to implement this as the customer location with a Notes backend. Then I will know more if this also works at a real data center location.

16

@arnold79

Thanks. Very helpful information.

1 Like