Elasticsearch Watcher - socket_timeout_exception when trying to send a .pdf email attachment generated from a Dashboard

Elastic Stack Elasticsearch
1

Hello there!

I have been tasked with setting up a Watcher that will email a pdf attachment that is generated from a dashboard.

After creating the Watcher and letting it run, I get repeated errors that point to a 'socket_timeout_exception'. Here is the full error:

{
  "watch_id": "85aec3de-a51e-4b40-a83c-6bf2274b2e2d",
  "node": "d3ZAQkBRSee0ccAnA527Vg",
  "state": "executed",
  "@timestamp": "2024-09-27T17:21:42.428Z",
  "user": "Adam Meindel",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2024-09-27T17:12:36.607Z"
    },
    "last_checked": "2024-09-27T17:21:42.428Z",
    "last_met_condition": "2024-09-27T17:21:42.428Z",
    "actions": {
      "email_admin": {
        "ack": {
          "timestamp": "2024-09-27T17:12:36.607Z",
          "state": "awaits_successful_execution"
        },
        "last_execution": {
          "timestamp": "2024-09-27T17:21:42.428Z",
          "successful": false,
          "reason": ""
        }
      }
    },
    "execution_state": "executed",
    "version": -1
  },
  "trigger_event": {
    "type": "schedule",
    "triggered_time": "2024-09-27T17:21:42.428Z",
    "schedule": {
      "scheduled_time": "2024-09-27T17:21:42.232Z"
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "metadata": {
    "name": "[ADAM] Google Maps Breakdown Report",
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2024-09-27T17:21:42.428Z",
    "execution_duration": 45038,
    "input": {
      "type": "none",
      "status": "success",
      "payload": {}
    },
    "condition": {
      "type": "always",
      "status": "success",
      "met": true
    },
    "actions": [
      {
        "id": "email_admin",
        "type": "email",
        "status": "failure",
        "error": {
          "root_cause": [
            {
              "type": "connect_timeout_exception",
              "reason": "Connect to kibana.cvges01-prod.amwell.systems:5601 [kibana.cvges01-prod.amwell.systems/3.17.237.206, kibana.cvges01-prod.amwell.systems/3.18.97.233, kibana.cvges01-prod.amwell.systems/18.116.219.112] failed: Connect timed out"
            }
          ],
          "type": "connect_timeout_exception",
          "reason": "Connect to kibana.cvges01-prod.amwell.systems:5601 [kibana.cvges01-prod.amwell.systems/3.17.237.206, kibana.cvges01-prod.amwell.systems/3.18.97.233, kibana.cvges01-prod.amwell.systems/18.116.219.112] failed: Connect timed out",
          "caused_by": {
            "type": "socket_timeout_exception",
            "reason": "Connect timed out"
          }
        }
      }
    ]
  },
  "messages": []
}

Here is the Watcher code:

{
  "trigger": {
    "schedule": {
      "interval": "24h"
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "attachments": {
          "google_maps_breakdown_report.pdf": {
            "reporting": {
              "url": "https://kibana.my-companys-cluster.systems/s/sandbox/api/reporting/generate/printablePdfV2?jobParams=%28browserTimezone%3AAmerica%2FNew_York%2Clayout%3A%28dimensions%3A%28height%3A1211.9886474609375%2Cwidth%3A1439.0908203125%29%2Cid%3Apreserve_layout%29%2ClocatorParams%3A%21%28%28id%3ADASHBOARD_APP_LOCATOR%2Cparams%3A%28dashboardId%3A%279af94ff1-2cf5-4b9b-8f44-7dbced496359%27%2CpreserveSavedFilters%3A%21t%2CtimeRange%3A%28from%3Anow-24h%2Fh%2Cto%3Anow%29%2CuseHash%3A%21f%2CviewMode%3Aview%29%29%29%2CobjectType%3Adashboard%2Ctitle%3A%27Google%20Maps%20Breakdown%27%2Cversion%3A%278.13.4%27%29",
              "retries": 40,
              "interval": "15s",
              "auth": {
                "basic": {
                  "username": "service",
                  "password": "::es_redacted::"
                }
              }
            }
          }
        },
        "to": [
          "'Recipient Name <adam.meindel@xyz.com>'"
        ],
        "subject": "Google Maps Breakdown Report"
      }
    }
  }
}

The 'url' link was generated from Dashboard -> Share -> PDF Reports -> Advanced options -> Copy POST URL

I've created a Watcher that sends just an email without the attachment logic, so I've ruled out the possibility of a cluster-side SMTP issue.

Any advice or suggestions are welcome.

Thank you in advance!

Adam

2

So it is timing out while contacting kibana for the report to attach to the email? I believe that can happen if the report generation is slow. Is there anything you could do to speed up this report? Is there anything interesting in the kibana or elasticsticsearch logs at the time it fails?

3

Hi, Keith. Thanks for responding.

The pdf report, when manually generated, doesn't take much time at all (less than 20 seconds).

I'm not sure exactly how to search the kibana and/or elasticsearch logs or what to even look for. I'm not seeing anything in Discover that would represent those. Admittedly, the documentation for this isn't very clear at all.

4

Are you using an Elastic-hosted cluster? Or is this self-hosted (machines that you/your company manage)?

5

Hi, Keith. The company I work for is self hosting via AWS EKS clusters (one for prod and one for the lower environment).

6

I ended up figuring out what the issue was.

It looks like there wasn't any DNS lookup (I'm guessing) for resolving the Kibana URL. I substituted the IP number and used http instead of https and the report executed and was emailed successfully:

{
  "trigger": {
    "schedule": {
      "interval": "24h"
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "attachments": {
          "google_maps_breakdown_report.pdf": {
            "reporting": {
              "url": "http://10.11.12.13/s/sandbox/api/reporting/generate/printablePdfV2?jobParams=%28browserTimezone%3AAmerica%2FNew_York%2Clayout%3A%28dimensions%3A%28height%3A1211.9886474609375%2Cwidth%3A1439.0908203125%29%2Cid%3Apreserve_layout%29%2ClocatorParams%3A%21%28%28id%3ADASHBOARD_APP_LOCATOR%2Cparams%3A%28dashboardId%3A%279af94ff1-2cf5-4b9b-8f44-7dbced496359%27%2CpreserveSavedFilters%3A%21t%2CtimeRange%3A%28from%3Anow-24h%2Cto%3Anow%29%2CuseHash%3A%21f%2CviewMode%3Aview%29%29%29%2CobjectType%3Adashboard%2Ctitle%3A%27Google%20Maps%20Breakdown%27%2Cversion%3A%278.13.4%27%29",
              "retries": 5,
              "interval": "15s",
              "auth": {
                "basic": {
                  "username": "service",
                  "password": "::es_redacted::"
                }
              }
            }
          }
        },
        "to": [
          "'Recipient Name <adam.meindel@xyz.com>'"
        ],
        "subject": "Google Maps Breakdown Report 2"
      }
    }
  }
}

Hopefully this will help someone in the future that is experiencing a similar issue.

Thanks!

1 Like