Functionbeat lambda timeout on aws elasticsearch connection

mnlth · April 28, 2020, 11:30pm

I have functionbeat lambda deployed with cloudwatch triggers which works fine. When lambda streams logs to elasticsearch, I get
Connecting to backoff(elasticsearch( https://escluster.es.amazonaws.com:9200 ))

using https with no certs as the lambda and elasticsearch are on same aws account with required IAM permissions
elasticserach host is appended with default port and not using just the dns

Anyone has got this working?

Luca_Belluccini · April 29, 2020, 12:17am

Using HTTP or HTTPS? How can you use HTTPS without certificates?
If you're not using certificates, use http://escluster.es.amazonaws.com:9200).

What is the Elasticsearch output you've configured?

You have to specify some parameters in the output as detailed here.

More in general, all the steps are here.

Try to run Functionbeat in debug (see documentation) and to view the logs, go to the the monitoring area of the AWS Lambda console and view the CloudWatch log group for the function.

mnlth · April 29, 2020, 2:29am

Thanks.
On the HTTPS configuration, documentation refers to " The ssl.certificate and ssl.key settings are ONLY needed if Elasticsearch is configured to require client based PKI authentication"
Since the elasticsearch doesnt require client auth, it has been simplified with output looking like below:

output.elasticsearch: hosts: ["escluster.es.amazonaws.com"] protocol: https

Also I have log level set to debug, which gives me no additional trace than what I have mentioned earlier

Luca_Belluccini · April 29, 2020, 6:54am

Do you use update in Functionbeat to update the function?

If you are using self signed certificates you have to add the certificate authorities, with the ca doit.

kvch · April 29, 2020, 2:41pm

Could you please share your configuration formatted using </>? Also, please share the debug logs. Without seeing exactly what Functionbeat does, it is hard to debug it.

mnlth · April 30, 2020, 3:23am

Config looks like

functionbeat.provider.aws.functions:
  - name: fb-cloudwatch
    enabled: true
    type: cloudwatch_logs
        description: "lambda function for cloudwatch logs"

        # Execution role of the function.
        role: arn:aws:iam::xxxxxxx:role/lambda.functionbeat.role

        virtual_private_cloud:
          security_group_ids: 
            - sg-xxxxxx
          subnet_ids:
            - subnet-xxxxxx
            - subnet-xxxxxx

        triggers:
          - log_group_name: /aws/lambda/samplelambda-loggrp

    setup.template.name: "test-functionbeat"
    setup.template.pattern: "test-functionbeat-*"

    output.elasticsearch:
      hosts: ["escluster.es.amazonaws.com"]
      protocol: https
      index: "test-functionbeat-%{+yyyy.MM.dd}"

    logging.level: debug

and logs. also interesting is that the lambda refers to functionbeat home as my local machine path instead

2020-04-28T18:35:31.083+10:00
2020-04-28T08:35:31.082Z	DEBUG	[processors]	processing/processors.go:186	Publish event: {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"@timestamp": "2020-04-28T08:35:19.822Z",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"@metadata": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"beat": "functionbeat",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"type": "_doc",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"version": "7.6.0"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"agent": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"type": "functionbeat",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"ephemeral_id": "5cf01230-e370-4287-98f8-8801aa5f3f61",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"hostname": "xxx.xxx.xx.x",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"id": "2732cbf7-50e4-4db4-9296-81b3c35879ce",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"version": "7.6.0"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"id": "35414986534184988452148185403028499011667930268676063232",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"message_type": "DATA_MESSAGE",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"subscription_filters": [
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"fnb-fb-cloudwatch-stack-fnbfbcloudwatchSFawslambda"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
],
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"message": "START RequestId: 6ca9d2b3-4ff9-4597-bb96-9fa72c9c4142 Version: $LATEST\n",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"owner": "867558745853",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"log_stream": "2020/04/28/[$LATEST]ce44eef4547b45a1830e1807525faf2b",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"host": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"name": "169.254.37.149"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"ecs": {
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"version": "1.4.0"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
},
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"log_group": "/aws/lambda/testlambda"
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
}
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.083+10:00
"message": "2020-04-28T08:35:27.054Z\t6ca9b3-4ff9-4597-bb96-9f2c9c4142\tINFO\t Request successful, status code : 200\r\n",
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77
.
.
.
.
.
.
.

2020-04-28T18:35:31.083+10:00
"message": "REPORT RequestId: 6ca9b3-4ff9-4597-bb96-9fa2c4142\tDuration: 7229.51 ms\tBilled Duration: 7300 ms\tMemory Size: 128 MB\tMax Memory Used: 90 MB\t\n",
.
.
.
.
.
2020-04-28T18:35:31.093+10:00
2020-04-28T08:35:31.093Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(https://escluster.es.amazonaws.com:9200))
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:31.093+10:00
2020-04-28T08:35:31.093Z DEBUG [elasticsearch] elasticsearch/client.go:733 ES Ping(url=https://escluster.es.amazonaws.com:9200)
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.069+10:00
END RequestId: 9c17f3ad-d51d-4618-a82c-31def762b2b3
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.069+10:00
REPORT RequestId: 9c17f3ad-d51d-4618-a82c-31def762b2b3 Duration: 3003.19 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 86 MB Init Duration: 408.68 ms
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.069+10:00
2020-04-28T08:35:34.068Z 9c17f3ad-d51d-4618-a82c-31def762b2b3 Task timed out after 3.00 seconds
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.625+10:00
2020-04-28T08:35:34.625Z INFO instance/beat.go:622 Home path: [/Users/xxxxx/functionbeat-7.6.0-darwin-x86_64] Config path: [/Users/xxxx/functionbeat-7.6.0-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.625+10:00
2020-04-28T08:35:34.625Z DEBUG [beat] instance/beat.go:674 Beat metadata path: /tmp/meta.json
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.625+10:00
2020-04-28T08:35:34.625Z INFO instance/beat.go:630 Beat ID: 2732cbf7-50e4-4db4-9296-81b3c35879ce
2020/04/28/[$LATEST]5cb581a3370e4bc38f7dbd74e3c86c77

2020-04-28T18:35:34.629+10:00
2020-04-28T08:35:34.628Z INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp

mnlth · April 30, 2020, 9:40am

With some network issues resolved, I was able to get past above error. But only to see a different error while connecting to es

Failed to connect to backoff(elasticsearch(https://escluster.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license from the /_xpack endpoint, xxx.xxx.xxx.93 requires the default distribution of Elasticsearch. Please make the endpoint accessible to xxx.xxx.xxx.93 so it can verify the license.: unauthorized access, could not connect to the xpack endpoint, verify your credentials

Christian_Dahlqvist · April 30, 2020, 12:41pm

Functionbeat requires the default distribution of Elasticsearch and is therefore not compatible with AWS Elasticsearch service. It should however work with Elastic’s Elasticsearch service.

mnlth · May 4, 2020, 2:41am

Thanks, that looks like it. I'm working with AWS ES which doesn't have the complete ES distribution

Rene_Benner · May 12, 2020, 8:44pm

I don't mean to stray from the initial topic here too far, but how does one deploy the ca.crt alongside the function then? This is documented literally nowhere, as the deployed function also seems to keep local paths.

I'm also suffering from the infamous need of a ca certificate in the function..

Luca_Belluccini · May 12, 2020, 10:30pm

Hello @Rene_Benner

I agree the documentation is lacking this part and it will be tackled.
We have a Github issue for it https://github.com/elastic/beats/issues/17885

The workaround is detailed at https://github.com/elastic/beats/issues/16969#issue-579644830, which consists in bundling the certificate in the package before uploading.

Hope it helps.

system · June 2, 2020, 12:30pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.