Help writing Watcher

Hello,

I have troubles writing a watcher which will watch (obviously) a field in my logs. I have a field called "sev" who has a number for each log, and I want to monitore it. For Example if I have sev = 1. It'll send me an e-mail. For now I have this, but it doesn't seem to work. Can you help me?

   {
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "syslog*"
        ],
        "types": [],
        "body": {
          "query": {
            "match": {
              "sev": "1"
            }
          }
        },
        "filter" : {
                "range": {
                  "@timestamp": {
                    "from": "{{ctx.trigger.scheduled_time}}||-5m",
                    "to": "{{ctx.trigger.triggered_time}}"
                }
            }
        }
      }
    }
  },
  
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "to": [
          "adrien.lechantre@orditech.be"
        ],
        "subject": "Field sev contains 1 !"
      }
    }
  }
}

I modified the JSON a bit and now it seems to accept but I don't receive any emails. Any Ideas?

{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "syslog*"
        ],
        "types": [],
        "body": {
          "query": {
            "match": {
              "sev": "1"
            }
          }
        }
      }
    }
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "to": [
          "adrien.lechantre@orditech.be"
        ],
        "subject": "Field sev contains 1 !"
      }
    }
  }
}

Hey,

My biggest ask and recommendation here would be to read this lengthy blog post, which introduces you into the black art (ok not so black, not even hard after a bit of looking at it) how to write and debug watches. This includes tips like running the execute watch API and reducing the feedback loop.

Afrer reading (and maybe even be able to interpret the execute watch api output) you could just paste a few more things, like the execute watch api output or the watcher history into this ticket (or into a gist), so we can solve your problem together.

Thanks!

--Alex

Hello Alex,

I've read it and the watch seems to be running, but I have error when it wants to send e-mails. I've read the log error in Elasticsearch and it says that Elastic needs an account, I searched over the net about that and the doc says I need to put this in the elasticsearch.yml file:

watcher.actions.email.service.account:
exchange_account:
    profile: outlook
    email_defaults:
        from: <email address of service account> 
    smtp:
        auth: true
        starttls.enable: true
        host: <your exchange server>
        port: 587
        user: <email address of service account> 
        password: <password>

But I still receive error from elastic:

Suppressed: java.lang.IllegalArgumentException: unknown setting [watcher.actions.email.service.account.outlook_account.smtp.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

But nowhere it says that I have to install plugin for emails. Is it make sense?

Ho, and if I don't put in the elasticsearch.yml file, I have this error :slight_smile:

[2018-02-14T15:47:18,215][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [in5oHa7] failed to execute action [j/email_admin]
java.lang.IllegalArgumentException: no accounts of type [email] configured. Please set up an account using the [xpack.notification.email] settings

You may have read outdated documention. The current one is here and uses xpack.notification.email.account as its prefix.

--Alex

Thank you ! Now it's trying to send the mail, but fails to connect:

[2018-02-14T16:24:41,737][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [in5oHa7] failed to execute action [j/email_admin]
javax.mail.MessagingException: failed to send email with subject [Field sev contains 1 !] via account [exchange_account]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:112) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:104) ~[?:?]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:84) ~[?:?]
        at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:156) ~[x-pack-core-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:492) ~[x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:322) ~[x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$7(ExecutionService.java:426) ~[x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:580) [x-pack-watcher-6.2.0.jar:6.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [elasticsearch-6.2.0.jar:6.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_161]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_161]
Caused by: com.sun.mail.util.MailConnectException: Couldn't connect to host, port: mx1.orditech.be, 587; timeout 120000
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2118) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:712) ~[?:?]
        at javax.mail.Service.connect(Service.java:366) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.lambda$executeConnect$2(Account.java:166) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.watcher.notification.email.Account.executeConnect(Account.java:165) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.send(Account.java:124) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:110) ~[?:?]
        ... 11 more
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_161]
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.8.0_161]
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.8.0_161]
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.8.0_161]
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_161]
        at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_161]
        at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.8.0_161]
        at java.net.Socket.connect(Unknown Source) ~[?:1.8.0_161]
        at com.sun.mail.util.WriteTimeoutSocket.connect(WriteTimeoutSocket.java:113) ~[?:?]
        at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:329) ~[?:?]
        at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:238) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2084) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:712) ~[?:?]
        at javax.mail.Service.connect(Service.java:366) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.lambda$executeConnect$2(Account.java:166) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.watcher.notification.email.Account.executeConnect(Account.java:165) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.send(Account.java:124) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:110) ~[?:?]
        ... 11 more

conf file:

xpack.notification.email.account:
    exchange_account:
        profile: outlook
        email_defaults:
            from: adrien.lechantre@orditech.be
        smtp:
            auth: true
            starttls.enable: true
            host: mx1.orditech.be
            port: 587
            user: adrien.lechantre@orditech.be
            password: <mypassword>

I think my conf file is correct, or maybe I'm missing something .. :confused:

there seems to be a connection issue to connect to that system. have you tried to connect from the system where the watch is executed on to connect to that host? It looks as if it might be firewalled.

Correction: Actually there is a message Connection refused: connect - which means there is nothing that watcher can connect to... maybe you need to use port 25 instead?

I thought first that it would've blocked it, but... It works .. Thanks a lot, Now I'm getting spam, is there a way to only send the mail once ?

it's all about the query. You may want to add more filters to your query, like only querying the last 5 minutes and search for special contents.

Also you can configure throttle period, see https://www.elastic.co/guide/en/x-pack/6.2/actions.html#actions-ack-throttle

A big thank you, I have one last question, is there a way to prevent the query e-mailing if it already e-mailed when it detected a field=1 in the same log ?

I mean, if I receive a log with the sev field = 1, It'll e-mail me once. But will it stop or countinue ? If so, can we stop the watcher to send me email about the log he already detected?

Hey,

this is usually what the timestamp filter is for. Otherwise you would need to mark each document that you may have found, and this is not what watcher is made for.

Hope this helps!

--Alex

I'll work this out Alex, thank you for your help !

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.