Elasticsearch Watcher error while trying to send email attachment, dashboard.pdf

Hi,

I have created a watcher alert from the advanced option which sends dashboard.pdf as an email attachment when the triggering condition is met. Now when the criteria is matching (threshold is exceeded) then it is throwing error as below in the watcher output.

          "root_cause": [
            {
              "type": "connect_timeout_exception",
              "reason": "Connect to mydomainname.com:443 [mydomainname.com/XX.X.XXX.XXX] failed: Connect timed out"
            }
          ],
          "type": "connect_timeout_exception",
          "reason": "Connect to mydomainname.com:443 [mydomainname.com/XX.X.XXX.XXX] failed: Connect timed out",
          "caused_by": {
            "type": "socket_timeout_exception",
            "reason": "Connect timed out"

Below is found from Elasticsearch log.

[2022-03-29T11:39:54,682][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [node-1] failed to execute action [test_watcher_1_last10mins_gte5_tran_dt_accord_sof/email_admin]
org.apache.http.conn.ConnectTimeoutException: Connect to mydomainname.com:443 [mydomainname.com/XX.X.XXX.XXX] failed: Connect timed out
....
....
Caused by: java.net.SocketTimeoutException: Connect timed out

Below is the watcher script.

{
  "trigger": {
    "schedule": {
      "interval": "6m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "my_index"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
		    "bool": {
			"must": [
			{
             "bool": {
              "minimum_should_match": 1,
              "should": [
                {
                  "match_phrase": {
                    "partner.keyword": "RXGTY"
                  }
                },
                {
                  "match_phrase": {
                    "partner.keyword": "VGHUT"
                  }
                }
              ]
			  }
			  },
                {
                  "match": {
                    "state.keyword": {"query": "Fail"}
                  }
                },
                {
                  "match": {
                    "ops.keyword": {"query": "api_name"}
                  }
                }
              ],
              "filter": {
                "range": {
                  "datetime": {
                    "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                    "lte": "{{ctx.trigger.scheduled_time}}",
                    "format": "strict_date_optional_time||epoch_millis"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "if (ctx.payload.hits.total >= params.threshold) { return true; } return false;",
      "lang": "painless",
      "params": {
        "threshold": 1
      }
    }
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "attachments": {
          "dashboard.pdf": {
            "reporting": {
              "url": "https://mydomainname.com/api/reporting/generate/printablePdf?jobParams= ..removing the rest portion of the url for security reason",
			  "auth": {"type":"basic","username":"elastic","password":"pass"}
            }
          },
          "data.yml": {
            "data": {
              "format": "yaml"
            }
          }
        },
		"from": "from_email@xyz.com",
        "to": [
          "to_email_name <to_email@abc.com>"
        ],
        "subject": "Elastic Watcher : Alert 1",
        "body": {
          "text": "Too many error in the system, see attached data."
        }
      }
    }
  },
  "transform": {
    "script": {
      "source": "HashMap result = new HashMap(); result.result = ctx.payload.hits.total; return result;",
      "lang": "painless",
      "params": {
        "threshold": 1
      }
    }
  }
}

Our elastic stack version is 7.11.1 and the license is activated, basic stack security is enabled.

Note that, when I have tried the same from local kibana (7.10.1), where the trial license is activated, there this alerting action is working perfectly. Also note that, in my local stack, the security feature is not enabled.

Please help!!

Regards,
Souvik

Two things. First, in order to have alerting with reporting up and running you need a Gold license. Second, can you reach the endpoint from the node that is executing that watch? It looks as if the connection times out.

We have the Platinum license. But further looking into the issue, I found that it might be the issue with chromium as while we were trying to download the pdf report from the dashboard, getting the below error.

Chromium error

And FYI, kibana resides in an ubuntu 18.04 server and seems this "Chromium" error is causing the above-mentioned error.

Could you please help us to resolve the "Error: Error spawning Chromium browser!" error?

what platform is this and can you take a closer look at the kibana logs?

Hi @spinscale , The platform is Kibana, getting the chromium error while trying to download the dashboard pdf i.e.,

dashboards > share > PDF Reports > Generate PDF

Below is the error log found from kibana.log

{"type":"response","@timestamp":"2022-04-01T07:43:15+00:00","tags":[],"pid":27755,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/printablePdf","method":"post","headers":{"connection":"upgrade","host":"mydomainname.com","content-length":"831","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"99\", \"Google Chrome\";v=\"99\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36","kbn-version":"7.11.1","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://mydomainname.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://mydomainname.com/app/dashboards","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"XXX.X.X.X","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36","referer":"https://mydomainname.com/app/dashboards"},"res":{"statusCode":200,"responseTime":84,"contentLength":9},"message":"POST /api/reporting/generate/printablePdf 200 84ms - 9.0B"}
{"type":"log","@timestamp":"2022-04-01T07:43:15+00:00","tags":["info","plugins","reporting","esqueue","queue-worker"],"pid":27755,"message":"l1evmqzi0lez2fee5d7ho139 - Job marked as claimed: /.reporting-2022-03-27/_doc/l1g4cmxs0lez2fee5d6248or"}
{"type":"log","@timestamp":"2022-04-01T07:43:15+00:00","tags":["info","plugins","reporting","esqueue","queue-worker"],"pid":27755,"message":"l1evmqzi0lez2fee5d7ho139 - Starting job"}
{"type":"log","@timestamp":"2022-04-01T07:43:15+00:00","tags":["info","plugins","reporting","printable_pdf","execute-job","l1g4cmxs0lez2fee5d6248or","l1g4cmxs0lez2fee5d6248or","browser-driver"],"pid":27755,"message":"Creating browser page driver"}
{"type":"log","@timestamp":"2022-04-01T07:43:15+00:00","tags":["error","plugins","reporting","printable_pdf","execute-job","l1g4cmxs0lez2fee5d6248or","l1g4cmxs0lez2fee5d6248or"],"pid":27755,"message":"Error: Error spawning Chromium browser!\n    at Observable._subscribe (/usr/share/kibana/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js:174:24)"}
{"type":"log","@timestamp":"2022-04-01T07:43:15+00:00","tags":["error","plugins","reporting","esqueue","queue-worker","error"],"pid":27755,"message":"l1evmqzi0lez2fee5d7ho139 - Failure occurred on job l1g4cmxs0lez2fee5d6248or: Error: Error spawning Chromium browser!\n    at Observable._subscribe (/usr/share/kibana/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js:174:24)"}
{"type":"log","@timestamp":"2022-04-01T07:43:15+00:00","tags":["warning","plugins","reporting","esqueue","queue-worker"],"pid":27755,"message":"l1evmqzi0lez2fee5d7ho139 - Failing job l1g4cmxs0lez2fee5d6248or"}

The log is referring to the same error which I got from Kibana UI.

How to fix this issue? Please guide.

sorry for not being clear. platform as in operating system as well as hardware architecture.

Hi @spinscale , We have 5 VMs for Elastic Stack. 3 are for 3 Elasticsearch nodes and 1 is for Kibana and 1 is for Logstash. Below is the configurations.

server config

Please let me know if you need further information.

Hi @spinscale , Did you get an opportunity to look into this?

Hi, anyone from #elastic, Could you please look into this issue? As it's a major issue currently we are facing and due to this, we are unable to configure certain watchers for the client.

Did you carefully read through this?

1 Like

Thank you so much @stephenb , for the reference. It helped! after installing nss packages in the VM, Reporting is now working.

1 Like

Hi @stephenb , Thanks for your suggestion, it helped us to resolve the chromium issue, and now I can download the PDF Reports from Generate PDF option.
Though the chromium issue is now resolved, still I am unable to send dashboard.pdf using watcher. Getting the same error both from watcher and Elasticsearch log. See below.

[2022-05-05T10:53:07,723][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [node-1] failed to execute action [test_watcher_1_last10mins_gte5_tran_dt_accord_sof/email_admin]
org.apache.http.conn.ConnectTimeoutException: Connect to mydomainname.com:443 [mydomainname.com/xx.x.xxx.xxx] failed: Connect timed out
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.10.jar:4.5.10]
        at org.elasticsearch.xpack.watcher.common.http.HttpClient.lambda$execute$1(HttpClient.java:240) ~[x-pack-watcher-7.13.0.jar:7.13.0]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:33) ~[x-pack-core-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.common.http.HttpClient.execute(HttpClient.java:240) ~[x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.notification.email.attachment.ReportingAttachmentParser.requestReportGeneration(ReportingAttachmentParser.java:258) ~[x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.notification.email.attachment.ReportingAttachmentParser.toAttachment(ReportingAttachmentParser.java:164) ~[x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.notification.email.attachment.ReportingAttachmentParser.toAttachment(ReportingAttachmentParser.java:53) ~[x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:63) [x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:166) [x-pack-core-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:534) [x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:320) [x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:421) [x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:627) [x-pack-watcher-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.13.0.jar:7.13.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: java.net.SocketTimeoutException: Connect timed out
        at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
        at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]
        at java.net.Socket.connect(Socket.java:645) ~[?:?]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.10.jar:4.5.10]
        ... 25 more

Unable to find any solution till now. Please guide.

Looks like you have a connectivity issue between Elasticsearch and Kibana you will need to debug that.

Assuming you are following the docs and using the full pdf linin the watcher.

Also did you include the credentials in the watcher?

Can you provide the watcher so we can take a look?

Hi @stephenb , it was a connectivity issue. The Kibana port was not accessible from Elasticsearch node. We have resolved it and now the watcher can generate pdf report and send it via email to us.

Many thanks for the suggestion!! It helped me to think in that direction. :slight_smile:

Regards,
Souvik

1 Like