# Watcher in x-pack returns NullPointerException

**URL:** <https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259>\
**Category:** Elasticsearch\
**Created:** [September 12, 2016, 7:56am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259 "2016-09-12T07:56:26Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [September 12, 2016, 7:56am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/1 "2016-09-12T07:56:26Z")

</div>

Hi!

I would like to ask question related to watcher in x-pack.

My environment is

x-pack -5.0.0 alpha5  
elasticsearch 5.0.0alpha5

I get below error when watcher is executed.

```
[2016-09-12 16:43:12,322][ERROR][xpack.watcher.actions.email] [v1_l7Fa] failed to execute action [log_errors2/email_administrator]
java.lang.NullPointerException
	at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111)
	at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:70)
	at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:64)
	at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85)
	at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:109)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:404)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:280)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:476)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:450)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

```

Watcher API(execute) shows that condition : status : **success** but email is not sent to root@localhost.localadmin. This is the same server as where elasticsearch is installed.

I have checked that below command is success and mail was sent successfully. So my mailserver **postfix** is working.

`echo "test" | mail root@localhost.localadmin`

What is this NullPointException is exactly referring to?

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [September 15, 2016, 7:40am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/2 "2016-09-15T07:40:21Z")

</div>

I was wondering , what is below line doing??

`at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111)`

Does anybody know?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 14, 2016, 7:24am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/3 "2016-10-14T07:24:08Z")

</div>

Hey,

can you share your configuration as well as the email sending part of your watch? Does this still happen with rc1?

--Alex

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 9, 2016, 3:45am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/4 "2016-11-09T03:45:50Z")

</div>

Yes it does.

My sending part is below. Also this is a environment where proxy exists.

```
"actions" : {
  "email_administrator" : {
    "from" : "test@localhost.localdomain",
   "to" : "y-watanabe@localhost.localdomain",
    "subject" : "Encountered errors",
   "body" : "Too many error in the system, see attached data",
   "attachments" : {
     "attached_data": {
       "data" : {"format" : "json"}
     },
     "priority" : "high"
  }
}

```

Is there any way to change the log level of watcher?  
I need to know which parameter is the root cause.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 9, 2016, 8:12am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/5 "2016-11-09T08:12:35Z")

</div>

Can you also share your `elasticsearch.yml` configuration for xpack?

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 17, 2016, 2:01am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/6 "2016-11-17T02:01:53Z")

</div>

@spinscale

Sorry for the late reply.

Below is the elasticsearch.yml  
I changed the ip information due to security reasons.

```
 [root@10028 elasticsearch]# cat elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
# インデックス領域を定義する
path.data: /data/elasticsearch/data
#
# Path to log files:
#
# ログ領域を定義する
#path.logs: /path/to/logs
path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
#node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

# General XPack setting
xpack.http.proxy.host: 192.168.1.1
xpack.http.proxy.port: 8080

# XPack watcher setting
xpack.watcher.enabled: false
xpack.notification.email.account:
  work:
    profile: standard
    smtp:
      host: 192.168.1.34
      port: 25
```

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 17, 2016, 2:05am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/7 "2016-11-17T02:05:52Z")

</div>

Below is the latest error log.

```
[2016-11-09T12:51:39,988][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [I2nQ1Xs] failed to execute action [fw/email_administrator]
java.lang.NullPointerException
        at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:68) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:62) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:161) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:63) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
```

---

<div class="post-metadata">

**Author:** ![arnold79](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arnold79/32/141473_2.png) [@arnold79](https://discuss.elastic.co/u/arnold79)\
**Post date:** [November 20, 2016, 4:56pm UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/8 "2016-11-20T16:56:06Z")

</div>

Hi,

As mentioned above. I've also having java.lang.NullPointerExceptions with latest X-pack release 5.0.0 on CentOS7. Also my postfix config is working and currently no firewall is running.

@YuWatanabe @spinscale Did you already found a solution for this ?

Failed to execute action [cluster\_health\_watch/send\_mail]  
java.lang.NullPointerException  
at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111) ~[x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:68) ~[x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:62) ~[x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85) ~[x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:161) [x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:496) [x-pack-5.0.0.jar:5.0.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444) [elasticsearch-5.0.0.jar:5.0.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0\_111]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0\_111]  
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_111]

---

<div class="post-metadata">

**Author:** ![arnold79](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arnold79/32/141473_2.png) [@arnold79](https://discuss.elastic.co/u/arnold79)\
**Post date:** [November 20, 2016, 7:14pm UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/9 "2016-11-20T19:14:58Z")

</div>

After upgrading the Elastic stack (elasticsearch, kibana and x-pack) to level 5.0.1 mail functionality is working as a charm 🙂

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 21, 2016, 8:40am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/10 "2016-11-21T08:40:59Z")

</div>

Hey Yu,

this cannot be your configuration, given that watcher is disabled? Or is that the only difference?

In addition your actions snippet is also not valid for watcher. Can you please share the whole watch?

With fixing your configuration and adding a valid watch I was again able to send an email.

You can increase logging by configuring `config/log4j2.properties` and setting

```auto
logger.action.name = org.elasticsearch.xpack.watcher
logger.action.level = debug

```

@arnold79 are you able to share your configuration and your watch as well so I can try to reproduce with 5.0.0 and ensure everything is fixed as it should be?

--Alex

---

<div class="post-metadata">

**Author:** ![arnold79](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arnold79/32/141473_2.png) [@arnold79](https://discuss.elastic.co/u/arnold79)\
**Post date:** [November 21, 2016, 9:23am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/11 "2016-11-21T09:23:42Z")

</div>

My e-mail configuration was working after implementing 5.0.1 for elasticsearch, kibana and x-pack components.

Here is my configuration (sample code) . Hopefully this helps

**Watcher trigger/input/condition/actions**

```
PUT _xpack/watcher/watch/cluster_health_watch
{
  "trigger": {
    "schedule": { "interval": "10s" }
  },
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9200,
        "path": "/_cluster/health",
        "auth": {
          "basic": {
            "username": "elastic",
            "password": "updateme"
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.status": { "eq":"yellow"}
    }
  },
  "actions": {
    "send_mail": {
      "email": {
        "to": "username@gmail.com",
        "subject": "Elastic Stack Alert",
        "body": "Cluster status is Yellow"
      }
    }
  }
}

GET .watcher-history*/_search
{
  "sort": [
    { "result.execution_time": "desc" }
    ]
}

DELETE _xpack/watcher/watch/cluster_health_watch

```

**elasticsearch.yml configuration**

```
xpack.notification.email.account:
  local_account:
    email_defaults:
      from: xpack@server2.nl
    profile: standard
    smtp:
      host: 127.0.0.1
      port: 25

```

**e-mail configuration on CentOS7**

`Default postfix and mail relyhost configured only !!!`

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 21, 2016, 9:49am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/12 "2016-11-21T09:49:15Z")

</div>

Below is the watcher definition. Nothing special really.

```
PUT _xpack/watcher/watch/fw
{
  "metadata": {
    "color" : "red"
  },
  "trigger" : {
    "schedule" : {
      "interval" : "10m" 
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : "poc-fw-gaibu-ext",
        "body" : {
          "query" : {
            "bool" : {
              "must" : [
                 {"range" : {"@timestamp" : {"gte" : "now-31d", "lte" : "now"}}}
              ]
            }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : {"ctx.payload.hits.total" : {"gt" : 0 } }
  },
  "actions" : {
    "email_administrator" : {
      "email" : {
        "from": "log.analyze.kensa.bsj@bridgestone.com",
        "to" : "y-watanabe@creationline.com",
        "subject" : "Encountered errors",
        "body" : "Too many error in the system, see attached data",
        "attachments": {	
          "attached_data" : {
            "data" : {"format" : "json" }
          }
        },
        "priority" : "high"
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 21, 2016, 9:54am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/13 "2016-11-21T09:54:34Z")

</div>

@arnold79 @spinscale

My problem is occurring at customer's POC site. I do not have a access but I will check error still occurs after upgrade to 5.0.1.

If still happens , I will check the debug log.

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 21, 2016, 10:01am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/14 "2016-11-21T10:01:07Z")

</div>

@arnold79

Do you have any proxy setting enabled in your elasticsearch server ?  
I know smtp is not related to proxy but would like to know any other major difference other than firewall.

---

<div class="post-metadata">

**Author:** ![arnold79](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arnold79/32/141473_2.png) [@arnold79](https://discuss.elastic.co/u/arnold79)\
**Post date:** [November 21, 2016, 10:21am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/15 "2016-11-21T10:21:26Z")

</div>

> [@YuWatanabe](#):
>
> setting enabled in your elasticsearch server ?

i.e. my elasticsearch.yml is default and only updated with the x-pack additions (in this case).

No proxy enabled,but I has NAT based internet access (VirtualBox default) and yes the local firewall of CENTOS7 is running with a exclude for smtp service.

```
 firewall-cmd --permanent --add-service=smtp
firewall-cmd --reload

```

Next Wednesday I have to implement this as the customer location with a Notes backend. Then I will know more if this also works at a real data center location.

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [November 21, 2016, 10:35am UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/16 "2016-11-21T10:35:14Z")

</div>

@arnold79

Thanks. Very helpful information.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:40pm UTC](https://discuss.elastic.co/t/watcher-in-x-pack-returns-nullpointerexception/60259/17 "2017-07-06T13:40:05Z")

</div>