Watcher Alert mail Issue

Elastic Stack Elasticsearch
1

Hi Team,

I have created new watcher

PUT /_watcher/watch/cluster_health_watch
{
"trigger" : {
"schedule" : { "interval" : "30s" }
},
"input" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : port,
"path" : "/_cluster/health"
}
}
},
"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "green" }
}
},
"actions" : {
"send_email" : {
"email" : {
"to" : "ganeshbabu.ramamoorthy@company.com",
"subject" : "Cluster Status Health",
"body" : "Cluster status is Green"
}
}
}
}

When I checked watcher history i am getting the following results,

{
"took": 61,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 27,
"max_score": 1,
"hits": [
{
"_index": ".watch_history-2015.11.03",
"_type": "watch_record",
"_id": "cluster_health_watch_0-2015-11-03T07:18:09.701Z",
"_score": 1,
"_source": {
"watch_id": "cluster_health_watch",
"state": "execution_not_needed",
"trigger_event": {
"type": "schedule",
"triggered_time": "2015-11-03T07:18:09.701Z",
"schedule": {
"scheduled_time": "2015-11-03T07:18:09.395Z"
}
},
"input": {
"http": {
"request": {
"scheme": "http",
"host": "localhost",
"port": port,
"method": "get",
"path": "/_cluster/health",
"params": {},
"headers": {}
}
}
},
"condition": {
"compare": {
"ctx.payload.status": {
"eq": "green"
}
}
},
"messages": [],
"result": {
"execution_time": "2015-11-03T07:18:09.701Z",
"execution_duration": 105,
"input": {
"type": "http",
"status": "success",
"payload": {},
"http": {
"request": {
"host": "localhost",
"port": port,
"scheme": "http",
"method": "get",
"path": "/_cluster/health"
},
"status_code": 401
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"ctx.payload.status": null
}
}
},
"actions": []
}
}
},

Why ctx.payload.status become "null" and action block shown as empty.

But I can see the count is increasing based on the interval(30s).

Why I didn't any alert mail from the server.

Is watcher will only work for negative scenario but I created watcher with positive scenario

will it work?

Please help me to resolve this issue.

I took the below link as reference to create watcher

https://www.elastic.co/guide/en/watcher/current/watch-cluster-status.html

Regards,
Ganeshbabu R

2

I'm by no way an expert in this but you gave the port the value port.
Wouldn't it now look for localhost:port ?
I once tried making it localhost2:9200 and it still showed up in Kibana but it never did anything of use.

So maybe you should give it a proper port.

3

Thanks for your reponse @SuperPringles

I tried by setting up the host as "localhost" but still "Payload" block returns no response and the status becomes "null"

Sample Watcher"
PUT /_watcher/watch/cluster_health_watch1
{
"trigger" : {
"schedule" : { "interval" : "30s" }
},
"input" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "/_cluster/health"
}
}
},
"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "green" }
}
},
"actions" : {
"send_email" : {
"email" : {
"to" : "ganeshbabu.ramamoorthy@company.com",
"subject" : "Cluster Status Health",
"body" : "Cluster status is Green"
}
}
}
}

Sample output:-
{
"_index": ".watch_history-2015.11.04",
"_type": "watch_record",
"_id": "cluster_health_watch1_0-2015-11-04T09:49:45.617Z",
"_score": 2.6563215,
"_source": {
"watch_id": "cluster_health_watch1",
"state": "execution_not_needed",
"trigger_event": {
"type": "schedule",
"triggered_time": "2015-11-04T09:49:45.617Z",
"schedule": {
"scheduled_time": "2015-11-04T09:49:45.137Z"
}
},
"input": {
"http": {
"request": {
"scheme": "http",
"host": "localhost",
"port": 9200,
"method": "get",
"path": "/_cluster/health",
"params": {},
"headers": {}
}
}
},
"condition": {
"compare": {
"ctx.payload.status": {
"eq": "green"
}
}
},
"messages": [],
"result": {
"execution_time": "2015-11-04T09:49:45.617Z",
"execution_duration": 54,
"input": {
"type": "http",
"status": "success",
"payload": {},
"http": {
"request": {
"host": "localhost",
"port": 9200,
"scheme": "http",
"method": "get",
"path": "/_cluster/health"
},
"status_code": 401
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"ctx.payload.status": null
}
}
},
"actions": []
}
}
},

Why status_code:- 401? Is it because of not given authentication details.

Can any one help me to resolve this/

Thanks,
Ganeshbabu R

4

This would be an example of how I make my watcher. As you can see I don't add authentication details either. Is it possible that your setup doesn't have enough permissions?

 curl -XPUT 'http://localhost:9200/_watcher/watch/cluster_health_watch3' -d '{
      "trigger" : {
        "schedule" : { "interval" : "10s" } 
      },
      "input" : {
        "http" : {
          "request" : {
           "host" : "localhost",
           "port" : 9200,
           "path" : "/_cluster/health"
          }
        }
      },
      "condition" : {
        "compare" : {
          "ctx.payload.status" : { "eq" : "green" }
        }
      },
      "actions" : {
        "send_email" : {
          "email" : {
            "to" : "myemail@gmail.com",
            "subject" : "Cluster Status Warning",
            "body" : "Cluster status is YELLOW"
          }
        }
      }
    }'

I also notice how your "state" returns "execution_not_needed"
This means it did not deem it necessary to execute it. This is shown when it compared but it wasn't the same value.

I get this when I look for a "red" status when my target is in the "yellow". I do get a payload however

   "payload": {
      "cluster_name": "elasticsearch",
      "status": "yellow",
      "timed_out": false,
      "number_of_nodes": 1,
      "number_of_data_nodes": 1,
      "active_primary_shards": 17,
      "active_shards": 17,
      "relocating_shards": 0,
      "initializing_shards": 0,
      "unassigned_shards": 17,
      "delayed_unassigned_shards": 0,
      "number_of_pending_tasks": 0,
      "number_of_in_flight_fetch": 0
    },

I suspect it can't show you this information because it doesn't have access to them which is probably why you're getting a 401. Like I said, maybe there is something wrong with the permissions?

I hope this helps you? I'm pretty much new to ubuntu and elastic all together but I want to help you either way :smile:

5

Thanks for your response @SuperPringles

Without authentication I cannot create watcher using curl command. It will show authentication exception
"error":"AuthenticationException[missing authentication token for REST request [/_watcher/watch/cluster_health_watch1

because I installed shield plugin in elasticsearch, So Its expecting authentication details to do some operations.

But still i get the same above error "Payload" block returns no response and the status becomes "null"

Similarly I created the same type of watcher with search type request.

PUT /_watcher/watch/item_watch
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"indices": [
"item"
],
"body": {
"query": {
"match": {
"ITEM_ID": 2746068
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"email_admin": {
"email": {
"to": "ganeshbabu.ramamoorthy@company.com",
"subject": "{{ctx.watch_id}} executed",
"body": "{{ctx.watch_id}} executed with {{ctx.payload.hits.total}} hits"
}
}
}
}

But for the search type request watcher I am getting proper response like getting alert mail.

I don't know why It's not working for http request. I am just following the documentation given in the elasticsearch watcher. I need to find it out help from others.

Thanks,
Ganeshbabu R

6

I haven't used Shield so I can only imagine but is it possible that Shield is blocking HTTP requests as part of its protection?

7

Hi Ganeshbabu,

If you are querying against the same cluster that Watcher is installed on, you should be using the 'search' input instead of the HTTP input. The search input will execute the query locally, and won't require username/password config.

If you are querying a remote cluster over HTTP(s), and that remote cluster is protected by Shield, you will have to include authentication information as part of the Watcher HTTP input.

https://www.elastic.co/guide/en/watcher/current/input.html#input-search

Thanks,
Steve

1 Like
8

Not entirely as I had suspected but I was pretty close :smile:

9

Thanks for your response @skearns

Suppose If I am querying against the same cluster over HTTP and the watcher is installed on.
My understanding is

  1. the watcher will not finds the cluster health.
  2. Only external elasticsearch clusters will help the HTTP to find the cluster health

Is it right?

Regards,
Ganeshbabu R

10

Hi all,

Below is the Watcher Email configuration from elasticsearch.yml,

################################## Watcher #################################

watcher.actions.email.service.account:
es_account:
profile: standard
smtp:
auth: false
starttls.enable: true
host: smarthost.enterprise.com
port: 25
user:
password:

I have created watcher to find the cluster_health,

PUT /_watcher/watch/cluster_health_watch
{
"trigger" : {
"schedule" : { "interval" : "60s" }
},
"input" : {
"http" : {
"request" : {
"host" : "10.7.146.26",
"port" : 9200,
"path" : "/_cluster/health",
"auth": {
"basic": {
"username": "esadmin",
"password": "password"
} } } } },
"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "green" }
} },
"actions" : {
"email_administrator" : {
"email" : {
"to" : "ganeshbabu.ramamoorthy.ap@company.com",
"subject" : "Cluster Status Health",
"body" : "Cluster status is Green",
"attach_data" : true,
"priority" : "high"
} } } }

I configured watcher email setup in master node only and below is the error response from master log during watcher execution.

Note:- Still I can get watcher alert mail from the server even though the error showing in master log.

[2015-12-15 01:21:06,085][ERROR][watcher.actions.email ] [dayrhed001_DEV_MASTER] failed to execute action [cluster_health_watch/email_administrator]
org.elasticsearch.watcher.actions.email.service.EmailException: failed to send email with subject [Cluster Status Health] via account [es_account]
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:86)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:78)
at org.elasticsearch.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:67)
at org.elasticsearch.watcher.actions.ActionWrapper.execute(ActionWrapper.java:104)
at org.elasticsearch.watcher.execution.ExecutionService.executeInner(ExecutionService.java:379)
at org.elasticsearch.watcher.execution.ExecutionService.execute(ExecutionService.java:271)
at org.elasticsearch.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.AuthenticationFailedException: No authentication mechanisms supported by both server and client
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:818)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
at javax.mail.Service.connect(Service.java:364)
at org.elasticsearch.watcher.actions.email.service.Account.send(Account.java:99)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:84)

Please guide me to resolve this error.

Thanks
Ganeshbabu R