Configure Slack/Email Action on Watch failing - yml issue

Joff · January 17, 2018, 9:04am

Hi,

I'm trying to configure Slack Action in Watcher but every time I update the elasticsearch.yml as per https://www.elastic.co/guide/en/x-pack/current/actions-slack.html#configuring-slack my node gets offline

I edited elasticsearch.yml as:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack:
  account: Admin
    monitoring:
      url: https://hooks.slack.com/services/mywebhookxxxxxxxx

I also tried to changed the account to something else or to add an email address linked to the account outcome is always the same.

If I check logs from /var/logs/elasticsearch I doesn't show any activity, just that the node is closed

If I run /usr/share/elasticsearch/bin$ sudo systemctl status elasticsearch.service then I got:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-01-17 08:46:59 UTC; 978ms ago
Docs: http://www.elastic.co
Process: 3491 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 3491 (code=exited, status=1/FAILURE)

Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)
Jan 17 08:46:59 master-0 elasticsearch[3491]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: mapping values are not allowed here
Jan 17 08:46:59 master-0 elasticsearch[3491]:  in 'reader', line 13, column 15:
Jan 17 08:46:59 master-0 elasticsearch[3491]:         monitoring:
Jan 17 08:46:59 master-0 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 17 08:46:59 master-0 systemd[1]: elasticsearch.service: Unit entered failed state.
Jan 17 08:46:59 master-0 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Now from digging this I kind of understand there might be space issue in my yaml next to 'monitoring'

If I edit the yml such as

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack:
  account: Admin
  monitoring:
    url: https://hooks.slack.com/services/mywebhookxxxxxxxx

then I got:

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-01-17 08:59:18 UTC; 3s ago
Docs: http://www.elastic.co
Process: 4304 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 4304 (code=exited, status=1/FAILURE)

Jan 17 08:59:18 master-0 systemd[1]: Started Elasticsearch.
Jan 17 08:59:18 master-0 elasticsearch[4304]: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000099f30000, 1712128000, 0) failed; error='Cannot allocate memory' (errn
Jan 17 08:59:18 master-0 elasticsearch[4304]: #
Jan 17 08:59:18 master-0 elasticsearch[4304]: # There is insufficient memory for the Java Runtime Environment to continue.
Jan 17 08:59:18 master-0 elasticsearch[4304]: # Native memory allocation (mmap) failed to map 1712128000 bytes for committing reserved memory.
Jan 17 08:59:18 master-0 elasticsearch[4304]: # An error report file with more information is saved as:
Jan 17 08:59:18 master-0 elasticsearch[4304]: # /tmp/hs_err_pid4304.log
Jan 17 08:59:18 ELK2master-0 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 17 08:59:18 master-0 systemd[1]: elasticsearch.service: Unit entered failed state.
Jan 17 08:59:18 master-0 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Did I break more the yml or what's next fix?

spinscale · January 17, 2018, 9:39am

Hey,

I think the problem here is the unquoted @ sign, and also a wrong configuration/indentation, can you try

xpack.notification.slack.account.monitoring.url: SLACK_HOOK_HERE

and see if that works?

Joff · January 17, 2018, 9:47am

yes thx already tried that, actually I was to update that before posting

Nevertheless after restarting my VMs (I run on Azure cloud envrioenment) the nodes are back IF the yml is edited such as:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack:
  account: Admin
  monitoring:
    url: https://hooks.slack.com/services/mywebhookxxxxxxxx

The problem now is that when I try to add a new watch in Kibana (v6.1 btw) and select Slack as an action I get the following error:

`

Watcher: [illegal_argument_exception] no accounts of type [slack] configured. Please set up an account using the [xpack.notification.slack] settings

`

But the [xpack.notification.slack] settings are updated in elasticsearch.yml

any idea?

Joff · January 17, 2018, 11:19am

@spinscale I now also tried now as you suggested:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack.account.monitoring.url: https://hooks.slack.com/services/mywebhookxxxxxxxx

I still have the error

Watcher: [illegal_argument_exception] no accounts of type [slack] configured. Please set up an account using the [xpack.notification.slack] settings

Also my cluster went in yellow state (not sure if related)

Any new suggestion?

spinscale · January 17, 2018, 12:03pm

what Elasticsearch version are you using? What Elasticsearch distribution did you install? And which OS are you running on?

Joff · January 17, 2018, 12:11pm

Elastic 6.1 on Ubuntu (vm set on Azure cloud)

I figured it out and it does work now as
xpack.notification.slack.account.monitoring.url: https://hooks.slack.com/services/mywebhookxxxxxxxx

I just needed to make sure that all Master and Data node have an updated yml

thx a lot for the help

(my cluster is still yellow though but I'll fix that as well)

Joff · January 17, 2018, 8:12pm

@spinscale I now have an issue with setting up the email action, perhaps you also have a tip to fix the syntax?(if that'sagain the problem)

Now my elasticsearch.yml looks like:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack.account.monitoring.url: https://hooks.slack.com/services/mywebhookxxxxxxxx
xpack.notification.email.account:
    gmail_account:
        profile: outlook
        smtp:
            auth: true
            starttls.enable: true
            host: smtp-mail.outlook.com
            port: 587
            user: account
            password: pw

But my problem is that when I go to Kibana and I add a new watch, then select Add new action ->Email
and then click on 'test fire an email now' I always get the same error:

Watcher: e.data is null

Error: e.data is null
value/<@http://xx.xxx.xx.xx:5601/bundles/kibana.bundle.js?v=16350:153:75612
processQueue@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:132456
scheduleProcessQueue/<@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:133349
$digest@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:144239
$apply@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:147007
done@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:100015
completeRequest@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:104697
requestError@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:105545

I tried with an Outlook and a gmail account, outcome is the same.
This also blocks the Slack actions which also get the same error.
When removing the email setting part from yml file then Slack action is working again.

So I am almost sure it is a syntax but where/what?

chrisronline · January 18, 2018, 2:06pm

HI @Joff,

This should trigger a network request to api/watcher/watch/execute which you can see in the Network tab in the Dev Tools.

Can you try again and paste the entire Request Payload and Response?

Thanks,
Chris

Joff · January 22, 2018, 12:26pm

Actually it is impossible to save a Watch with an email action, it gives the same error and do not save.

Also when adding the email seeting to my ealsticsearch.yml file same error are then showing up for slack actions.

removing the email settings in yml file fixes the slack action but back to starting point with email

Any clue what's next?

chrisronline · January 24, 2018, 7:46pm

For the affected watches (just one is fine), can you paste the response for:

GET .watches/_search/

Feel free to scrub any sensitive data. I'm only interested in the structure

system · February 21, 2018, 7:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.