Watcher in x-pack returns NullPointerException

YuWatanabe · September 12, 2016, 7:56am

Hi!

I would like to ask question related to watcher in x-pack.

My environment is

x-pack -5.0.0 alpha5
elasticsearch 5.0.0alpha5

I get below error when watcher is executed.

[2016-09-12 16:43:12,322][ERROR][xpack.watcher.actions.email] [v1_l7Fa] failed to execute action [log_errors2/email_administrator]
java.lang.NullPointerException
	at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111)
	at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:70)
	at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:64)
	at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85)
	at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:109)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:404)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:280)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:476)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:450)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Watcher API(execute) shows that condition : status : success but email is not sent to root@localhost.localadmin. This is the same server as where elasticsearch is installed.

I have checked that below command is success and mail was sent successfully. So my mailserver postfix is working.

echo "test" | mail root@localhost.localadmin

What is this NullPointException is exactly referring to?

YuWatanabe · September 15, 2016, 7:40am

I was wondering , what is below line doing??

at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111)

Does anybody know?

spinscale · October 14, 2016, 7:24am

Hey,

can you share your configuration as well as the email sending part of your watch? Does this still happen with rc1?

--Alex

YuWatanabe · November 9, 2016, 3:45am

Yes it does.

My sending part is below. Also this is a environment where proxy exists.

"actions" : {
  "email_administrator" : {
    "from" : "test@localhost.localdomain",
   "to" : "y-watanabe@localhost.localdomain",
    "subject" : "Encountered errors",
   "body" : "Too many error in the system, see attached data",
   "attachments" : {
     "attached_data": {
       "data" : {"format" : "json"}
     },
     "priority" : "high"
  }
}

Is there any way to change the log level of watcher?
I need to know which parameter is the root cause.

spinscale · November 9, 2016, 8:12am

Can you also share your elasticsearch.yml configuration for xpack?

YuWatanabe · November 17, 2016, 2:01am

@spinscale

Sorry for the late reply.

Below is the elasticsearch.yml
I changed the ip information due to security reasons.

 [root@10028 elasticsearch]# cat elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
# インデックス領域を定義する
path.data: /data/elasticsearch/data
#
# Path to log files:
#
# ログ領域を定義する
#path.logs: /path/to/logs
path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
#node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

# General XPack setting
xpack.http.proxy.host: 192.168.1.1
xpack.http.proxy.port: 8080

# XPack watcher setting
xpack.watcher.enabled: false
xpack.notification.email.account:
  work:
    profile: standard
    smtp:
      host: 192.168.1.34
      port: 25

YuWatanabe · November 17, 2016, 2:05am

Below is the latest error log.

[2016-11-09T12:51:39,988][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [I2nQ1Xs] failed to execute action [fw/email_administrator]
java.lang.NullPointerException
        at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:68) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:62) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85) ~[x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:161) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:63) [x-pack-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

arnold79 · November 20, 2016, 4:56pm

Hi,

As mentioned above. I've also having java.lang.NullPointerExceptions with latest X-pack release 5.0.0 on CentOS7. Also my postfix config is working and currently no firewall is running.

@YuWatanabe @spinscale Did you already found a solution for this ?

Failed to execute action [cluster_health_watch/send_mail]
java.lang.NullPointerException
at org.elasticsearch.xpack.notification.email.Account.send(Account.java:111) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:68) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.notification.email.EmailService.send(EmailService.java:62) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:85) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:161) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:496) [x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444) [elasticsearch-5.0.0.jar:5.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

arnold79 · November 20, 2016, 7:14pm

After upgrading the Elastic stack (elasticsearch, kibana and x-pack) to level 5.0.1 mail functionality is working as a charm

spinscale · November 21, 2016, 8:40am

Hey Yu,

this cannot be your configuration, given that watcher is disabled? Or is that the only difference?

In addition your actions snippet is also not valid for watcher. Can you please share the whole watch?

With fixing your configuration and adding a valid watch I was again able to send an email.

You can increase logging by configuring config/log4j2.properties and setting

logger.action.name = org.elasticsearch.xpack.watcher
logger.action.level = debug

@arnold79 are you able to share your configuration and your watch as well so I can try to reproduce with 5.0.0 and ensure everything is fixed as it should be?

--Alex

arnold79 · November 21, 2016, 9:23am

My e-mail configuration was working after implementing 5.0.1 for elasticsearch, kibana and x-pack components.

Here is my configuration (sample code) . Hopefully this helps

Watcher trigger/input/condition/actions

PUT _xpack/watcher/watch/cluster_health_watch
{
  "trigger": {
    "schedule": { "interval": "10s" }
  },
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9200,
        "path": "/_cluster/health",
        "auth": {
          "basic": {
            "username": "elastic",
            "password": "updateme"
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.status": { "eq":"yellow"}
    }
  },
  "actions": {
    "send_mail": {
      "email": {
        "to": "username@gmail.com",
        "subject": "Elastic Stack Alert",
        "body": "Cluster status is Yellow"
      }
    }
  }
}


GET .watcher-history*/_search
{
  "sort": [
    { "result.execution_time": "desc" }
    ]
}

DELETE _xpack/watcher/watch/cluster_health_watch

elasticsearch.yml configuration

xpack.notification.email.account:
  local_account:
    email_defaults:
      from: xpack@server2.nl
    profile: standard
    smtp:
      host: 127.0.0.1
      port: 25

e-mail configuration on CentOS7

Default postfix and mail relyhost configured only !!!

YuWatanabe · November 21, 2016, 9:49am

Below is the watcher definition. Nothing special really.

PUT _xpack/watcher/watch/fw
{
  "metadata": {
    "color" : "red"
  },
  "trigger" : {
    "schedule" : {
      "interval" : "10m" 
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : "poc-fw-gaibu-ext",
        "body" : {
          "query" : {
            "bool" : {
              "must" : [
                 {"range" : {"@timestamp" : {"gte" : "now-31d", "lte" : "now"}}}
              ]
            }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : {"ctx.payload.hits.total" : {"gt" : 0 } }
  },
  "actions" : {
    "email_administrator" : {
      "email" : {
        "from": "log.analyze.kensa.bsj@bridgestone.com",
        "to"  : "y-watanabe@creationline.com",
        "subject" : "Encountered errors",
        "body" : "Too many error in the system, see attached data",
        "attachments": {	
          "attached_data" : {
            "data" : {"format" : "json" }
          }
        },
        "priority" : "high"
      }
    }
  }
}

YuWatanabe · November 21, 2016, 9:54am

@arnold79 @spinscale

My problem is occurring at customer's POC site. I do not have a access but I will check error still occurs after upgrade to 5.0.1.

If still happens , I will check the debug log.

YuWatanabe · November 21, 2016, 10:01am

@arnold79

Do you have any proxy setting enabled in your elasticsearch server ?
I know smtp is not related to proxy but would like to know any other major difference other than firewall.

arnold79 · November 21, 2016, 10:21am

i.e. my elasticsearch.yml is default and only updated with the x-pack additions (in this case).

No proxy enabled,but I has NAT based internet access (VirtualBox default) and yes the local firewall of CENTOS7 is running with a exclude for smtp service.

 firewall-cmd --permanent --add-service=smtp
firewall-cmd --reload

Next Wednesday I have to implement this as the customer location with a Notes backend. Then I will know more if this also works at a real data center location.

YuWatanabe · November 21, 2016, 10:35am

@arnold79

Thanks. Very helpful information.

Topic		Replies	Views
Watchers are not working after elasticsearch migration 7.16.1 to 8.5.1 Elasticsearch elastic-stack-alerting	2	419	April 30, 2023
Watcher: Internal Server Error Elasticsearch	9	1609	June 11, 2018
Failed to execute action [email_watch/send _email] Elasticsearch	4	1740	May 20, 2018
Unable to send email using watcher on Xpack 6.2.1 Elasticsearch elastic-stack-alerting	10	644	June 5, 2018
Failed to execute watch - Timeout waiting for task Elasticsearch	2	1864	September 13, 2021

Watcher in x-pack returns NullPointerException

Related Topics