Watcher is Giving 401 Status Errors When Trying to Execute 'email' Action

Elastic Stack Elasticsearch
,
1

Hi all,

I am trying to set up a Watcher to automatically send an email with a PDF of a dashboard. I have been following the documentation here.

However I am getting an error when the 'email' action is executed. I can't seem to figure out what the issue is because I can generate the PDF reports manually.

Below is my Watcher set up:

PUT _watcher/watch/daily_api_report
{
  "trigger" : {
    "schedule": {
      "interval" : "2m"
    }
  },
  "actions" : {
    "email_admin" : {
      "email" : {
        "to" : "EMAIL_ADDRESS",
        "subject" : "API Daily Report",
        "attachments" : {
          "API_report.pdf" : {
            "reporting" : {
              "url" : "https://localhost:5601/api/reporting/generate/printablePdf?jobParams=(browserTimezone:Europe%2FLondon,layout:(dimensions:(height:670,width:1805),id:preserve_layout),objectType:dashboard,relativeUrls:!(%27%2Fapp%2Fkibana%23%2Fdashboard%2F14b07490-b761-11e9-b7d1-5b5065ed50ed%3F_g%3D(refreshInterval:(pause:!!t,value:0),time:(from:now-30d,to:now))%26_a%3D(description:!%27!%27,filters:!!(),fullScreenMode:!!f,options:(hidePanelTitles:!!f,useMargins:!!t),panels:!!((embeddableConfig:(),gridData:(h:15,i:!%271!%27,w:24,x:24,y:7),id:!%275e3c0670-b53f-11e9-b641-a7a464783daa!%27,panelIndex:!%271!%27,type:visualization,version:!%277.3.0!%27),(embeddableConfig:(),gridData:(h:7,i:!%272!%27,w:14,x:0,y:0),id:!%273f8ce2b0-b75d-11e9-b7d1-5b5065ed50ed!%27,panelIndex:!%272!%27,type:visualization,version:!%277.3.0!%27),(embeddableConfig:(),gridData:(h:15,i:!%273!%27,w:24,x:0,y:7),id:!%276cb16840-b539-11e9-b641-a7a464783daa!%27,panelIndex:!%273!%27,type:visualization,version:!%277.3.0!%27),(embeddableConfig:(),gridData:(h:7,i:!%274!%27,w:34,x:14,y:0),id:!%2774b31770-b53c-11e9-b641-a7a464783daa!%27,panelIndex:!%274!%27,type:visualization,version:!%277.3.0!%27)),query:(language:kuery,query:!%27!%27),timeRestore:!!t,title:!%27API%2BPerformance%2BMonitor%2B(For%2BVanraj)!%27,viewMode:view)%27),title:%27API%20Performance%20Monitor%20(For%20Vanraj)%27)",
              "retries" : 2,
              "interval" : "1m",
              "auth" : {
                "username" : "elastic",
                "password" : "my_password"
              }
            }
          }
        }
      }
    }
  }
}

Below is my Watcher error when simulated:

{
  "watch_id": "_inlined_",
  "node": "n03UOqaqQZyQg63LlCjMmw",
  "state": "executed",
  "user": "elastic",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2019-08-07T15:10:46.694Z"
    },
    "last_checked": "2019-08-07T15:10:46.695Z",
    "last_met_condition": "2019-08-07T15:10:46.695Z",
    "actions": {
      "email_admin": {
        "ack": {
          "timestamp": "2019-08-07T15:10:46.694Z",
          "state": "awaits_successful_execution"
        },
        "last_execution": {
          "timestamp": "2019-08-07T15:10:46.695Z",
          "successful": false,
          "reason": ""
        }
      }
    },
    "execution_state": "executed",
    "version": -1
  },
  "trigger_event": {
    "type": "manual",
    "triggered_time": "2019-08-07T15:10:46.694Z",
    "manual": {
      "schedule": {
        "scheduled_time": "2019-08-07T15:10:46.694Z"
      }
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "metadata": {
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2019-08-07T15:10:46.695Z",
    "execution_duration": 486,
    "input": {
      "type": "none",
      "status": "success",
      "payload": {}
    },
    "condition": {
      "type": "always",
      "status": "success",
      "met": true
    },
    "actions": [
      {
        "id": "email_admin",
        "type": "email",
        "status": "failure",
        "error": {
          "root_cause": [
            {
              "type": "exception",
              "reason": "Watch[_inlined_] reporting[API_report.pdf] Error response when trying to trigger reporting generation host[localhost], port[5601] method[POST], path[/api/reporting/generate/printablePdf], status[401]"
            }
          ],
          "type": "exception",
          "reason": "Watch[_inlined_] reporting[API_report.pdf] Error response when trying to trigger reporting generation host[localhost], port[5601] method[POST], path[/api/reporting/generate/printablePdf], status[401]"
        }
      }
    ]
  },
  "messages": []
}

I have configured the X-Pack security settings and I think that I have secured the reporting endpoints by following this.

Does anyone know what the issue is being caused by?

2

Additionally, here are my elasticsearch.yml and kibana.yml files:

Elasticsearch:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: ["/home/nathan/elasticsearch-backup"]
xpack.security.enabled : true
xpack.watcher.encrypt_sensitive_data : true
xpack.http.whitelist: ["*"]
xpack.http.ssl.truststore.path: /etc/elasticsearch/watcher-truststore.jks
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elasticsearch-server.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elasticsearch-server.p12
xpack.security.audit.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/elasticsearch-server.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/elasticsearch-server.p12
xpack.notification.email.account:
        exchange_account:
                profile: outlook
                email_defaults:
                        from: email@domain.com
                smtp:
                        auth: true
                        starttls.enable: true
                        host: server.domain.com
                        port: 587
                        user: email@domain.com

Kibana:

elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana-server.crt
server.ssl.key: /etc/kibana/kibana-server.key
elasticsearch.ssl.certificate: /etc/kibana/elasticsearch-server.crt
elasticsearch.ssl.key: /etc/kibana/elasticsearch-server.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-stack-authorities.pem" ]
xpack.security.encryptionKey: "key that is 32 characters long"

Information such as email address and encryptionKey obfuscated.

3

have you tried using curl with that username and password and the URL specified in the watch? Does that work? It looks as if there is an authorization issue when trying to access kibana to create the PDF from within the watch.

--Alex

4

Thank you for your reply Alex.

I have tried using curl and it seems to be a certificate issue:

Enter host password for user 'elastic':
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Do you know if it is the Kibana certificate or the Elasticsearch certificate that is causing an issue?

5

as you do not query elasticsearch, this would be kibana. There is an option in curl to ignore the unavailability of the issue cert, that you can use for debugging.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.