Getting error while creating Watcher Email Configuration

Hi,

I am the new one for ELK installation setup, Now I have completed ELK setup end to end.
for POC I have enabled platinum license for one month.

Now am trying to setup watcher email configuration, But I couldn't, getting an error and don't know where i start from.

please share if you have any document for watcher configuration.

</ elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-06-06 04:52:46 UTC; 19s ago
Docs: http://www.elastic.co
Process: 65609 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 65609 (code=exited, status=1/FAILURE)

Jun 06 04:52:39 metron4 systemd[1]: Started Elasticsearch.
Jun 06 04:52:39 metron4 systemd[1]: Starting Elasticsearch...
Jun 06 04:52:40 metron4 elasticsearch[65609]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 06 04:52:46 metron4 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 06 04:52:46 metron4 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 06 04:52:46 metron4 systemd[1]: elasticsearch.service failed.

elasticsearch.yml:

watcher.actions.email.service.account:
gmail:
profile: gmail
smtp:
auth: true
starttls.enable: true
host: smtp.gmail.com
port: 587
user: xxxxx@gmail.com
password: yyyyy >

can you please share the elasticsearch log output? That should contain an error message helping you out.

If you are using version 7.0, you need to use the keystore command to store the password, see https://www.elastic.co/guide/en/elastic-stack-overview/7.1/actions-email.html

[2019-06-07T18:36:52,382][ERROR][o.e.x.w.t.s.ExecutableScriptTransform] [metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_528b9adc-3136-45b8-806c-f723033dc392-2019-06-07T18:36:52.29911Z]
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
... 11 more
[2019-06-07T18:37:52,426][ERROR][o.e.x.w.t.s.ExecutableScriptTransform] [metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_35d89c0f-3826-464f-915f-4f8494f7bec5-2019-06-07T18:37:52.312431Z]
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
... 11 more

This is my elasticsearch log once i enabled mail configuration in elasticsearch.yml

########################################################
[2019-06-07T18:37:52,426][ERROR][o.e.x.w.t.s.ExecutableScriptTransform] [metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_35d89c0f-3826-464f-915f-4f8494f7bec5-2019-06-07T18:37:52.312431Z]
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
... 11 more
[2019-06-07T18:38:52,477][ERROR][o.e.x.w.t.s.ExecutableScriptTransform] [metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_fe52012d-9454-4dcb-b388-cc3921fd199a-2019-06-07T18:38:52.325557Z]
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
... 11 more
[2019-06-07T18:39:49,101][INFO ][o.e.n.Node ] [metron4] stopping ...
[2019-06-07T18:39:49,124][INFO ][o.e.x.w.WatcherService ] [metron4] stopping watch service, reason [shutdown initiated]
[2019-06-07T18:39:49,341][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [metron4] [controller/73153] [Main.cc@148] Ml controller exiting
[2019-06-07T18:39:49,342][INFO ][o.e.x.m.p.NativeController] [metron4] Native controller process has stopped - no new native processes can be started
[2019-06-07T18:39:49,935][INFO ][o.e.n.Node ] [metron4] stopped
[2019-06-07T18:39:49,935][INFO ][o.e.n.Node ] [metron4] closing ...
[2019-06-07T18:39:49,970][INFO ][o.e.n.Node ] [metron4] closed

And my service also stopped.

####################

‚óŹ elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2019-06-07 18:39:52 UTC; 3s ago
Docs: http://www.elastic.co
Process: 60463 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 60463 (code=exited, status=1/FAILURE)

Jun 07 18:39:51 metron4 elasticsearch[60463]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSet...ava:83)
Jun 07 18:39:52 metron4 elasticsearch[60463]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
Jun 07 18:39:52 metron4 elasticsearch[60463]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun 07 18:39:52 metron4 elasticsearch[60463]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jun 07 18:39:52 metron4 elasticsearch[60463]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jun 07 18:39:52 metron4 elasticsearch[60463]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Jun 07 18:39:52 metron4 elasticsearch[60463]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jun 07 18:39:52 metron4 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 07 18:39:52 metron4 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 07 18:39:52 metron4 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

elasticsearch.yml
###################

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: ELK-cluster

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: systemname

Add custom attributes to the node:

#node.attr.rack: r1

#node.master: false
#node.data: true

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: IP Address

Set a custom port for HTTP:

#http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.seed_hosts: ["IP Address"]

Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: ["IP Address"]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true

#xpack.security.enabled: true
#----------------------------------- Email Notification -------------------------

#xpack.notification.email.account:
#watcher.actions.email.service.account:

gmail:

profile: gmail

smtp:

auth: true

starttls.enable: true

host: smtp.gmail.com

port: 587

user: YYYY@gmail.com

password: XXXX

#"actions" : {

"send_email" : { YYYY@gmail.com

"email" : { YYYY@gmail.com

"to" : "YYYY@gmail.com",

"subject" : "Watcher Error Notification",

"body" : "{{ctx.payload.hits.total}} error logs found"

}

}

#}

#xpack.notification.email.html.sanitization.enabled: false

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

please take your time to properly format your log messages. This is pretty much unreadable and people will not bother taking a look. You can use markdown here for formatting. Thank you!

[quote="stalinbritto, post:4, topic:184495"]
[2019-06-07T18:39:49,101][INFO ][o.e.n.Node ] [metron4] stopping ...
[2019-06-07T18:39:49,124][INFO ][o.e.x.w.WatcherService ] [metron4] stopping watch service, reason [shutdown initiated]
[2019-06-07T18:39:49,341][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [metron4] [controller/73153] [Main.cc@148] Ml controller exiting
[2019-06-07T18:39:49,342][INFO ][o.e.x.m.p.NativeController] [metron4] Native controller process has stopped - no new native processes can be started
[2019-06-07T18:39:49,935][INFO ][o.e.n.Node ] [metron4] stopped
[2019-06-07T18:39:49,935][INFO ][o.e.n.Node ] [metron4] closing ...
[2019-06-07T18:39:49,970][INFO ][o.e.n.Node ] [metron4] closed[/quote]

The above indicates a clear and initiated shutdown of elasticsearch. The exception above that is more than two minutes old and will not lead to a shutdown.

Are you unable to start elasticsearch again? If so, can you share the log output from that start? Thanks.

[2019-06-07T18:37:52,426][ERROR][o.e.x.w.t.s.ExecutableScriptTransform] [metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_35d89c0f-3826-464f-915f-4f8494f7bec5-2019-06-07T18:37:52.312431Z]
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
... 11 more
[2019-06-07T18:39:49,101][INFO ][o.e.n.Node ] [metron4] stopping ...
[2019-06-07T18:39:49,124][INFO ][o.e.x.w.WatcherService ] [metron4] stopping watch service, reason [shutdown initiated]
[2019-06-07T18:39:49,341][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [metron4] [controller/73153] [Main.cc@148] Ml controller exiting
[2019-06-07T18:39:49,342][INFO ][o.e.x.m.p.NativeController] [metron4] Native controller process has stopped - no new native processes can be started
[2019-06-07T18:39:49,935][INFO ][o.e.n.Node ] [metron4] stopped
[2019-06-07T18:39:49,935][INFO ][o.e.n.Node ] [metron4] closing ...
[2019-06-07T18:39:49,970][INFO ][o.e.n.Node ] [metron4] closed

you repasted the same log message, I was asking from a log excerpt from the startup.

[ERROR]
[o.e.x.w.t.s.ExecutableScriptTransform] [metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_35d89c0f-3826-464f-915f-4f8494f7bec5-2019-06-07T18:37:52.312431Z]
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
... 11 more
[INFO ][o.e.n.Node ] [metron4] stopping ...
[INFO ][o.e.x.w.WatcherService ] [metron4] stopping watch service, reason [shutdown initiated]
INFO ][o.e.x.m.p.l.CppLogMessageHandler] [metron4] [controller/73153] [Main.cc@148] Ml controller exiting
[INFO ][o.e.x.m.p.NativeController] [metron4] Native controller process has stopped - no new native processes can be started
[INFO ][o.e.n.Node ] [metron4] stopped
[INFO ][o.e.n.Node ] [metron4] closing ...
[INFO ][o.e.n.Node ] [metron4] closed

[metron4] failed to execute [script] transform for [-8plsEZsRteVdd-3GYHXmQ_elasticsearch_cluster_status_35d89c0f-3826-464f-915f-4f8494f7bec5-2019-06-07T18:37:52.312431Z]
`Preformatted text`org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:1152) ~[?:?]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:49) ~[x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:23) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:505) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:309) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:410) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:605) [x-pack-watcher-7.1.0.jar:7.1.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:458) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0 && ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack != null) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ct ...:429) ~[?:?]
stopping ...
stopping watch service, reason [shutdown initiated]
Ml controller exiting
Native controller process has stopped - no new native processes can be started
stopped
closing ...
closed