Forbidden error when simulating an email report from watcher

Elastic Stack Kibana
1

I'm following the documentation here to create a watcher to automate some weekly reports. When I try to simulate the watcher I get the following error:

{
  "watch_id": "_inlined_",
  "node": "gdVCszd8SaO5-xSmFM2Gsw",
  "state": "executed",
  "user": "1251094200",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2023-10-25T17:07:20.027Z"
    },
    "last_checked": "2023-10-25T17:07:20.027Z",
    "last_met_condition": "2023-10-25T17:07:20.027Z",
    "actions": {
      "email_admin": {
        "ack": {
          "timestamp": "2023-10-25T17:07:20.027Z",
          "state": "awaits_successful_execution"
        },
        "last_execution": {
          "timestamp": "2023-10-25T17:07:20.027Z",
          "successful": false,
          "reason": ""
        }
      }
    },
    "execution_state": "executed",
    "version": -1
  },
  "trigger_event": {
    "type": "manual",
    "triggered_time": "2023-10-25T17:07:20.027Z",
    "manual": {
      "schedule": {
        "scheduled_time": "2023-10-25T17:07:20.027Z"
      }
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "metadata": {
    "name": "Weekly Error PDF",
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2023-10-25T17:07:20.027Z",
    "execution_duration": 22,
    "input": {
      "type": "none",
      "status": "success",
      "payload": {}
    },
    "condition": {
      "type": "always",
      "status": "success",
      "met": true
    },
    "actions": [
      {
        "id": "email_admin",
        "type": "email",
        "status": "failure",
        "error": {
          "root_cause": [
            {
              "type": "exception",
              "reason": "Watch[_inlined_] reporting[error_report.pdf] Error response when trying to trigger reporting generation host[healthsnap-blue-audit.kb.us-east-1.aws.found.io], port[9243] method[POST], path[/api/reporting/generate/printablePdfV2], response[status=[403], headers=[[date: [Wed, 25 Oct 2023 17:07:20 GMT]], [content-length: [35]], [content-type: [application/json; charset=UTF-8]], [x-found-handling-cluster: [61f0f42851a44539845f845dbd997a47]], [x-cloud-request-id: [_5ehbJ9tRTyJQUoW0C_DiQ]]], body=[{\"ok\":false,\"message\":\"Forbidden\"}\n]]"
            }
          ],
          "type": "exception",
          "reason": "Watch[_inlined_] reporting[error_report.pdf] Error response when trying to trigger reporting generation host[healthsnap-blue-audit.kb.us-east-1.aws.found.io], port[9243] method[POST], path[/api/reporting/generate/printablePdfV2], response[status=[403], headers=[[date: [Wed, 25 Oct 2023 17:07:20 GMT]], [content-length: [35]], [content-type: [application/json; charset=UTF-8]], [x-found-handling-cluster: [61f0f42851a44539845f845dbd997a47]], [x-cloud-request-id: [_5ehbJ9tRTyJQUoW0C_DiQ]]], body=[{\"ok\":false,\"message\":\"Forbidden\"}\n]]"
        }
      }
    ]
  },
  "messages": []
}

My configuration for the watcher is the following:

{
  "trigger" : {
    "schedule": {
      "weekly" : { "on" : "monday", "at" : "6:00" }
    }
  },
  "actions" : {
    "email_admin" : { 
      "email": {
        "to": "'Audit Team <audit@company.com>'",
        "subject": "Weekly Audit Errors Report",
        "attachments" : {
          "error_report.pdf" : {
            "reporting" : {
              "url": "https://myserver.aws.found.io:9243/api/reporting/generate/printablePdfV2?jobParams=%28browserTimezone%3AAmerica%2FNew_York%2Clayout%3A%28dimensions%3A%28height%3A1212%2Cwidth%3A1499%29%2Cid%3Apreserve_layout%29%2ClocatorParams%3A%21%28%28id%3ADASHBOARD_APP_LOCATOR%2Cparams%3A%28dashboardId%3A%276ca912a0-90db-11ec-a901-07b1f786920c%27%2CpreserveSavedFilters%3A%21t%2CtimeRange%3A%28from%3Anow-1w%2Cto%3Anow%29%2CuseHash%3A%21f%2CviewMode%3Aview%29%2Cversion%3A%277.17.9%27%29%29%2CobjectType%3Adashboard%2Ctitle%3A%27Audit%20Dashboard%27%2Cversion%3A%277.17.9%27%29", 
              "retries":3, 
              "interval":"30s", 
              "auth":{ 
                "basic":{
                  "username":"elastic",
                  "password":"****"
                }
              }
            }
          }
        }
      }
    }
  }
}

When I try to access the same URL via CURL it works using the basic auth. Any idea on what the issue could be.

2

Hi I used (in a 8.10.4 cluster) your template, and I could create a watcher from the Dev Console like this:

PUT _watcher/watch/test_email
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "attachments": {
          "error_report.pdf": {
            "reporting": {
              "url": "URL-Generated-From-Dashrboard-Share-Pop-Up",
              "retries": 3,
              "interval": "30s",
              "auth": {
                "basic": {
                  "username": "elastic",
                  "password": "password-here"
                }
              }
            }
          }
        },
        "to": ["'Jorge Sanz <my-work-email>'"],
        "subject": "Test Report"
      }
    }
  }
}

In a couple minutes the first report came successfully.

image
image530×656 57.5 KB

It is true that later I can't edit the watcher JSON because it fails to validate the auth section

image

And the same error shows when I try to simulate the action:

image
image1035×286 23.6 KB

So maybe this is related to your authentication issue.

I'll create an issue to report about this to the team.

3
4

Hi, thank you for taking the time to look into this. But I did try already with the put, and I still get the same error.

Screenshot 2023-10-27 at 10.37.52 AM
Screenshot 2023-10-27 at 10.37.52 AM2550×2382 705 KB

The issue that you are reporting is simple to fix. All you have to do is go into the watcher template and change the password from ::es_redacted:: to the actual password. I think it gets changed in the output for security reasons.

What I'm seeing on my side is that the watcher from Kibana cannot communicate with the cloud API endpoint. I wonder if I'm missing some configuration for the Kibana process to access that URL. We use a custom endpoint alias for our ES instances.

5

Have you checked your server logs? Maye there are more details there on the issue but I agree that it may be related with some connectivity problem between your instance and the destination endpoint.

Good luck!!

6

All I get from the server logs is just the Forbidden Access message and the point reference to the failing code. I opened a support ticket for it, but for now no answer on the issue.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.