Help writing Watcher

Adren · February 14, 2018, 1:25pm

Hello,

I have troubles writing a watcher which will watch (obviously) a field in my logs. I have a field called "sev" who has a number for each log, and I want to monitore it. For Example if I have sev = 1. It'll send me an e-mail. For now I have this, but it doesn't seem to work. Can you help me?

   {
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "syslog*"
        ],
        "types": [],
        "body": {
          "query": {
            "match": {
              "sev": "1"
            }
          }
        },
        "filter" : {
                "range": {
                  "@timestamp": {
                    "from": "{{ctx.trigger.scheduled_time}}||-5m",
                    "to": "{{ctx.trigger.triggered_time}}"
                }
            }
        }
      }
    }
  },
  
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "to": [
          "adrien.lechantre@orditech.be"
        ],
        "subject": "Field sev contains 1 !"
      }
    }
  }
}

Adren · February 14, 2018, 1:45pm

I modified the JSON a bit and now it seems to accept but I don't receive any emails. Any Ideas?

{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "syslog*"
        ],
        "types": [],
        "body": {
          "query": {
            "match": {
              "sev": "1"
            }
          }
        }
      }
    }
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "to": [
          "adrien.lechantre@orditech.be"
        ],
        "subject": "Field sev contains 1 !"
      }
    }
  }
}

spinscale · February 14, 2018, 1:55pm

Hey,

My biggest ask and recommendation here would be to read this lengthy blog post, which introduces you into the black art (ok not so black, not even hard after a bit of looking at it) how to write and debug watches. This includes tips like running the execute watch API and reducing the feedback loop.

Afrer reading (and maybe even be able to interpret the execute watch api output) you could just paste a few more things, like the execute watch api output or the watcher history into this ticket (or into a gist), so we can solve your problem together.

Thanks!

--Alex

Adren · February 14, 2018, 2:32pm

Hello Alex,

I've read it and the watch seems to be running, but I have error when it wants to send e-mails. I've read the log error in Elasticsearch and it says that Elastic needs an account, I searched over the net about that and the doc says I need to put this in the elasticsearch.yml file:

watcher.actions.email.service.account:
exchange_account:
    profile: outlook
    email_defaults:
        from: <email address of service account> 
    smtp:
        auth: true
        starttls.enable: true
        host: <your exchange server>
        port: 587
        user: <email address of service account> 
        password: <password>

But I still receive error from elastic:

Suppressed: java.lang.IllegalArgumentException: unknown setting [watcher.actions.email.service.account.outlook_account.smtp.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

But nowhere it says that I have to install plugin for emails. Is it make sense?

Ho, and if I don't put in the elasticsearch.yml file, I have this error

[2018-02-14T15:47:18,215][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [in5oHa7] failed to execute action [j/email_admin]
java.lang.IllegalArgumentException: no accounts of type [email] configured. Please set up an account using the [xpack.notification.email] settings

spinscale · February 14, 2018, 2:57pm

You may have read outdated documention. The current one is here and uses xpack.notification.email.account as its prefix.

--Alex

Adren · February 14, 2018, 3:26pm

Thank you ! Now it's trying to send the mail, but fails to connect:

[2018-02-14T16:24:41,737][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [in5oHa7] failed to execute action [j/email_admin]
javax.mail.MessagingException: failed to send email with subject [Field sev contains 1 !] via account [exchange_account]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:112) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:104) ~[?:?]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:84) ~[?:?]
        at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:156) ~[x-pack-core-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:492) ~[x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:322) ~[x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$7(ExecutionService.java:426) ~[x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:580) [x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [elasticsearch-6.2.0.jar:6.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_161]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_161]
Caused by: com.sun.mail.util.MailConnectException: Couldn't connect to host, port: mx1.orditech.be, 587; timeout 120000
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2118) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:712) ~[?:?]
        at javax.mail.Service.connect(Service.java:366) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.lambda$executeConnect$2(Account.java:166) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.watcher.notification.email.Account.executeConnect(Account.java:165) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.send(Account.java:124) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:110) ~[?:?]
        ... 11 more
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_161]
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.8.0_161]
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.8.0_161]
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.8.0_161]
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_161]
        at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_161]
        at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.8.0_161]
        at java.net.Socket.connect(Unknown Source) ~[?:1.8.0_161]
        at com.sun.mail.util.WriteTimeoutSocket.connect(WriteTimeoutSocket.java:113) ~[?:?]
        at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:329) ~[?:?]
        at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:238) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2084) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:712) ~[?:?]
        at javax.mail.Service.connect(Service.java:366) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.lambda$executeConnect$2(Account.java:166) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.watcher.notification.email.Account.executeConnect(Account.java:165) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.send(Account.java:124) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:110) ~[?:?]
        ... 11 more

conf file:

xpack.notification.email.account:
    exchange_account:
        profile: outlook
        email_defaults:
            from: adrien.lechantre@orditech.be
        smtp:
            auth: true
            starttls.enable: true
            host: mx1.orditech.be
            port: 587
            user: adrien.lechantre@orditech.be
            password: <mypassword>

I think my conf file is correct, or maybe I'm missing something ..

spinscale · February 14, 2018, 3:33pm

there seems to be a connection issue to connect to that system. have you tried to connect from the system where the watch is executed on to connect to that host? It looks as if it might be firewalled.

Correction: Actually there is a message Connection refused: connect - which means there is nothing that watcher can connect to... maybe you need to use port 25 instead?

Adren · February 14, 2018, 3:45pm

I thought first that it would've blocked it, but... It works .. Thanks a lot, Now I'm getting spam, is there a way to only send the mail once ?

spinscale · February 14, 2018, 3:50pm

it's all about the query. You may want to add more filters to your query, like only querying the last 5 minutes and search for special contents.

Also you can configure throttle period, see https://www.elastic.co/guide/en/x-pack/6.2/actions.html#actions-ack-throttle

Adren · February 14, 2018, 4:02pm

A big thank you, I have one last question, is there a way to prevent the query e-mailing if it already e-mailed when it detected a field=1 in the same log ?

I mean, if I receive a log with the sev field = 1, It'll e-mail me once. But will it stop or countinue ? If so, can we stop the watcher to send me email about the log he already detected?

spinscale · February 14, 2018, 8:14pm

Hey,

this is usually what the timestamp filter is for. Otherwise you would need to mark each document that you may have found, and this is not what watcher is made for.

Hope this helps!

--Alex

Adren · February 15, 2018, 7:36am

I'll work this out Alex, thank you for your help !

system · March 15, 2018, 7:36am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Watcher Timestamp showing in EPOCH Elasticsearch elastic-stack-alerting	4	908	October 1, 2021
How to call a filter/report in a watcher? Elasticsearch elastic-stack-alerting	6	1078	June 29, 2021
Need help with watcher to send the correct fields of result in the email Elasticsearch elastic-stack-alerting	6	1289	June 23, 2020
Watcher with field query error Elasticsearch elastic-stack-monitoring , elastic-stack-alerting	2	498	November 13, 2020
Creating a Watcher Rule Elasticsearch elastic-stack-alerting	2	328	April 9, 2020

Help writing Watcher

Related topics