Configure Slack/Email Action on Watch failing - yml issue

alerting

#1

Hi,

I'm trying to configure Slack Action in Watcher but every time I update the elasticsearch.yml as per https://www.elastic.co/guide/en/x-pack/current/actions-slack.html#configuring-slack my node gets offline

I edited elasticsearch.yml as:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack:
  account: Admin
    monitoring:
      url: https://hooks.slack.com/services/mywebhookxxxxxxxx

I also tried to changed the account to something else or to add an email address linked to the account outcome is always the same.

If I check logs from /var/logs/elasticsearch I doesn't show any activity, just that the node is closed

If I run /usr/share/elasticsearch/bin$ sudo systemctl status elasticsearch.service then I got:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-01-17 08:46:59 UTC; 978ms ago
Docs: http://www.elastic.co
Process: 3491 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 3491 (code=exited, status=1/FAILURE)

Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jan 17 08:46:59 master-0 elasticsearch[3491]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)
Jan 17 08:46:59 master-0 elasticsearch[3491]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: mapping values are not allowed here
Jan 17 08:46:59 master-0 elasticsearch[3491]:  in 'reader', line 13, column 15:
Jan 17 08:46:59 master-0 elasticsearch[3491]:         monitoring:
Jan 17 08:46:59 master-0 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 17 08:46:59 master-0 systemd[1]: elasticsearch.service: Unit entered failed state.
Jan 17 08:46:59 master-0 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Now from digging this I kind of understand there might be space issue in my yaml next to 'monitoring'

If I edit the yml such as

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack:
  account: Admin
  monitoring:
    url: https://hooks.slack.com/services/mywebhookxxxxxxxx

then I got:

‚óŹ elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-01-17 08:59:18 UTC; 3s ago
Docs: http://www.elastic.co
Process: 4304 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 4304 (code=exited, status=1/FAILURE)

Jan 17 08:59:18 master-0 systemd[1]: Started Elasticsearch.
Jan 17 08:59:18 master-0 elasticsearch[4304]: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000099f30000, 1712128000, 0) failed; error='Cannot allocate memory' (errn
Jan 17 08:59:18 master-0 elasticsearch[4304]: #
Jan 17 08:59:18 master-0 elasticsearch[4304]: # There is insufficient memory for the Java Runtime Environment to continue.
Jan 17 08:59:18 master-0 elasticsearch[4304]: # Native memory allocation (mmap) failed to map 1712128000 bytes for committing reserved memory.
Jan 17 08:59:18 master-0 elasticsearch[4304]: # An error report file with more information is saved as:
Jan 17 08:59:18 master-0 elasticsearch[4304]: # /tmp/hs_err_pid4304.log
Jan 17 08:59:18 ELK2master-0 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 17 08:59:18 master-0 systemd[1]: elasticsearch.service: Unit entered failed state.
Jan 17 08:59:18 master-0 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Did I break more the yml or what's next fix?


(Alexander Reelsen) #2

Hey,

I think the problem here is the unquoted @ sign, and also a wrong configuration/indentation, can you try

xpack.notification.slack.account.monitoring.url: SLACK_HOOK_HERE

and see if that works?


#3

yes thx already tried that, actually I was to update that before posting

Nevertheless after restarting my VMs (I run on Azure cloud envrioenment) the nodes are back IF the yml is edited such as:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack:
  account: Admin
  monitoring:
    url: https://hooks.slack.com/services/mywebhookxxxxxxxx

The problem now is that when I try to add a new watch in Kibana (v6.1 btw) and select Slack as an action I get the following error:

`

Watcher: [illegal_argument_exception] no accounts of type [slack] configured. Please set up an account using the [xpack.notification.slack] settings

`

But the [xpack.notification.slack] settings are updated in elasticsearch.yml

any idea?


#4

@spinscale I now also tried now as you suggested:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack.account.monitoring.url: https://hooks.slack.com/services/mywebhookxxxxxxxx

I still have the error

Watcher: [illegal_argument_exception] no accounts of type [slack] configured. Please set up an account using the [xpack.notification.slack] settings

Also my cluster went in yellow state (not sure if related)

Any new suggestion?


(Alexander Reelsen) #5

what Elasticsearch version are you using? What Elasticsearch distribution did you install? And which OS are you running on?


#6

Elastic 6.1 on Ubuntu (vm set on Azure cloud)

I figured it out and it does work now as
xpack.notification.slack.account.monitoring.url: https://hooks.slack.com/services/mywebhookxxxxxxxx

I just needed to make sure that all Master and Data node have an updated yml

thx a lot for the help

(my cluster is still yellow though but I'll fix that as well)


#7

@spinscale I now have an issue with setting up the email action, perhaps you also have a tip to fix the syntax?(if that'sagain the problem)

Now my elasticsearch.yml looks like:

cluster.name: MyCluster
node.name: master-0
discovery.zen.ping.unicast.hosts: ["master-0:9300","master-1:9300","master-2:9300"]
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 2
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.notification.slack.account.monitoring.url: https://hooks.slack.com/services/mywebhookxxxxxxxx
xpack.notification.email.account:
    gmail_account:
        profile: outlook
        smtp:
            auth: true
            starttls.enable: true
            host: smtp-mail.outlook.com
            port: 587
            user: account
            password: pw

But my problem is that when I go to Kibana and I add a new watch, then select Add new action ->Email
and then click on 'test fire an email now' I always get the same error:

Watcher: e.data is null

Error: e.data is null
value/<@http://xx.xxx.xx.xx:5601/bundles/kibana.bundle.js?v=16350:153:75612
processQueue@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:132456
scheduleProcessQueue/<@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:133349
$digest@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:144239
$apply@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:147007
done@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:100015
completeRequest@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:104697
requestError@http://xx.xxx.xx.xx:5601/bundles/commons.bundle.js?v=16350:29:105545

I tried with an Outlook and a gmail account, outcome is the same.
This also blocks the Slack actions which also get the same error.
When removing the email setting part from yml file then Slack action is working again.

So I am almost sure it is a syntax but where/what?


(Chris Roberson) #8

HI @Joff,

This should trigger a network request to api/watcher/watch/execute which you can see in the Network tab in the Dev Tools.

Can you try again and paste the entire Request Payload and Response?

Thanks,
Chris


#9

Actually it is impossible to save a Watch with an email action, it gives the same error and do not save.

Also when adding the email seeting to my ealsticsearch.yml file same error are then showing up for slack actions.

removing the email settings in yml file fixes the slack action but back to starting point with email

Any clue what's next?


(Chris Roberson) #10

For the affected watches (just one is fine), can you paste the response for:

GET .watches/_search/

Feel free to scrub any sensitive data. I'm only interested in the structure


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.