Functionbeat lambda timeout on aws elasticsearch connection

I have functionbeat lambda deployed with cloudwatch triggers which works fine. When lambda streams logs to elasticsearch, I get
Connecting to backoff(elasticsearch( https://escluster.es.amazonaws.com:9200 ))

  1. using https with no certs as the lambda and elasticsearch are on same aws account with required IAM permissions
  2. elasticserach host is appended with default port and not using just the dns

Anyone has got this working?

Using HTTP or HTTPS? How can you use HTTPS without certificates?
If you're not using certificates, use http://escluster.es.amazonaws.com:9200).

What is the Elasticsearch output you've configured?

You have to specify some parameters in the output as detailed here.

More in general, all the steps are here.


Try to run Functionbeat in debug (see documentation) and to view the logs, go to the the monitoring area of the AWS Lambda console and view the CloudWatch log group for the function.

Thanks.
On the HTTPS configuration, documentation refers to " The ssl.certificate and ssl.key settings are ONLY needed if Elasticsearch is configured to require client based PKI authentication"
Since the elasticsearch doesnt require client auth, it has been simplified with output looking like below:

output.elasticsearch: hosts: ["escluster.es.amazonaws.com"] protocol: https

Also I have log level set to debug, which gives me no additional trace than what I have mentioned earlier

Do you use update in Functionbeat to update the function?

If you are using self signed certificates you have to add the certificate authorities, with the ca doit.

Could you please share your configuration formatted using </>? Also, please share the debug logs. Without seeing exactly what Functionbeat does, it is hard to debug it. :slight_smile:

Config looks like

functionbeat.provider.aws.functions:
  - name: fb-cloudwatch
    enabled: true
    type: cloudwatch_logs
        description: "lambda function for cloudwatch logs"

        # Execution role of the function.
        role: arn:aws:iam::xxxxxxx:role/lambda.functionbeat.role

        virtual_private_cloud:
          security_group_ids: 
            - sg-xxxxxx
          subnet_ids:
            - subnet-xxxxxx
            - subnet-xxxxxx

        triggers:
          - log_group_name: /aws/lambda/samplelambda-loggrp

    setup.template.name: "test-functionbeat"
    setup.template.pattern: "test-functionbeat-*"

    output.elasticsearch:
      hosts: ["escluster.es.amazonaws.com"]
      protocol: https
      index: "test-functionbeat-%{+yyyy.MM.dd}"

    logging.level: debug

and logs. also interesting is that the lambda refers to functionbeat home as my local machine path instead

2020-04-28T18:35:31.083+10:00
2020-04-28T08:35:31.082Z	DEBUG	[processors]	processing/processors.go:186	Publish event: {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"@timestamp": "2020-04-28T08:35:19.822Z",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"@metadata": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"beat": "functionbeat",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"type": "_doc",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"version": "7.6.0"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"agent": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"type": "functionbeat",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"ephemeral_id": "5cf01230-e370-4287-98f8-8801aa5f3f61",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"hostname": "xxx.xxx.xx.x",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"id": "2732cbf7-50e4-4db4-9296-81b3c35879ce",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"version": "7.6.0"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"id": "35414986534184988452148185403028499011667930268676063232",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"message_type": "DATA_MESSAGE",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"subscription_filters": [
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"fnb-fb-cloudwatch-stack-fnbfbcloudwatchSFawslambda"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
],
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"message": "START RequestId: 6ca9d2b3-4ff9-4597-bb96-9fa72c9c4142 Version: $LATEST\n",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"owner": "867558745853",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"log_stream": "2020/04/28/[$LATEST]ce44eef4547b45a1830e1807525faf2b",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"host": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"name": "169.254.37.149"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"ecs": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"version": "1.4.0"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"log_group": "/aws/lambda/testlambda"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
}
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"message": "2020-04-28T08:35:27.054Z\t6ca9b3-4ff9-4597-bb96-9f2c9c4142\tINFO\t Request successful, status code : 200\r\n",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77
.
.
.
.
.
.
.

2020-04-28T18:35:31.083+10:00
"message": "REPORT RequestId: 6ca9b3-4ff9-4597-bb96-9fa2c4142\tDuration: 7229.51 ms\tBilled Duration: 7300 ms\tMemory Size: 128 MB\tMax Memory Used: 90 MB\t\n",
.
.
.
.
.
2020-04-28T18:35:31.093+10:00
2020-04-28T08:35:31.093Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(https://escluster.es.amazonaws.com:9200))
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.093+10:00
2020-04-28T08:35:31.093Z DEBUG [elasticsearch] elasticsearch/client.go:733 ES Ping(url=https://escluster.es.amazonaws.com:9200)
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.069+10:00
END RequestId: 9c17f3ad-d51d-4618-a82c-31def762b2b3
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.069+10:00
REPORT RequestId: 9c17f3ad-d51d-4618-a82c-31def762b2b3 Duration: 3003.19 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 86 MB Init Duration: 408.68 ms
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.069+10:00
2020-04-28T08:35:34.068Z 9c17f3ad-d51d-4618-a82c-31def762b2b3 Task timed out after 3.00 seconds
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.625+10:00
2020-04-28T08:35:34.625Z INFO instance/beat.go:622 Home path: [/Users/xxxxx/functionbeat-7.6.0-darwin-x86_64] Config path: [/Users/xxxx/functionbeat-7.6.0-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.625+10:00
2020-04-28T08:35:34.625Z DEBUG [beat] instance/beat.go:674 Beat metadata path: /tmp/meta.json
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.625+10:00
2020-04-28T08:35:34.625Z INFO instance/beat.go:630 Beat ID: 2732cbf7-50e4-4db4-9296-81b3c35879ce
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.629+10:00
2020-04-28T08:35:34.628Z INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp

With some network issues resolved, I was able to get past above error. But only to see a different error while connecting to es

Failed to connect to backoff(elasticsearch(https://escluster.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license from the /_xpack endpoint, xxx.xxx.xxx.93 requires the default distribution of Elasticsearch. Please make the endpoint accessible to xxx.xxx.xxx.93 so it can verify the license.: unauthorized access, could not connect to the xpack endpoint, verify your credentials

Functionbeat requires the default distribution of Elasticsearch and is therefore not compatible with AWS Elasticsearch service. It should however work with Elastic’s Elasticsearch service.

Thanks, that looks like it. I'm working with AWS ES which doesn't have the complete ES distribution :frowning:

I don't mean to stray from the initial topic here too far, but how does one deploy the ca.crt alongside the function then? This is documented literally nowhere, as the deployed function also seems to keep local paths.

I'm also suffering from the infamous need of a ca certificate in the function..

Hello @Rene_Benner

I agree the documentation is lacking this part and it will be tackled.
We have a Github issue for it https://github.com/elastic/beats/issues/17885

The workaround is detailed at https://github.com/elastic/beats/issues/16969#issue-579644830, which consists in bundling the certificate in the package before uploading.

Hope it helps.