How to integrate AWS Lambda with plugin Elastic

Andrea_Spacca · June 23, 2022, 7:18am

the ARNs have to be defined in both places: when deploying the Forwarder, so that proper permissions and triggers are set, and in the config file, so that you can define the output of a given input ARN

it's suggested to create and upload the config file to an s3 bucket, before deploying the Forwarder.
Also if you update the deployment, it is suggested to update the config file before, at least for adding new input ARNs

if you need to update the outputs only, you just update the config file and next execution of the lambda will use the updated config, with no needs for the Forwarder to be redeployed

Renato_Souza · June 24, 2022, 5:34am

okay, I successfully implanted. But there is some permission adjustment required, see:

That is right?

Andrea_Spacca · June 24, 2022, 5:56am

did you add the cloudwatch logs arn to the ElasticServerlessForwarderCloudWatchLogsEvents param when deploying the forwarder?

if you want to add an input after the first deployment you have to redeploy the forwarder: for that you follow the same step as creating a new function using the same "Application name". this will update the already deployed forwarder instead of creating a new one

beware that due to a bug in AWS CloudFormation, if you want to update the Events settings for the deployed Lambda, you will have to execute a deployment deleting the existing ones before actually apply the new updated ones.

this means that you have to update it twice: the first time emptying the following params:

      ElasticServerlessForwarderCloudWatchLogsEvents
      ElasticServerlessForwarderKinesisEvents
      ElasticServerlessForwarderS3SQSEvents
      ElasticServerlessForwarderSQSEvents

and the second time filling them again with the new values

if you updated the deployment adding the new value for ElasticServerlessForwarderCloudWatchLogsEvents without emptying it before, the deployment should have failed.

you can check in the cloudformation console on AWS or also see in the lambda details for the forwarder what are the permissions and the triggers set up for the lambda: the ones added with the update should not be present

Renato_Souza · June 24, 2022, 6:29am

I'm using terraform to do the deployment.

I did as follows:

provider "aws" {
  region = "us-east-1"  ## FILL WITH THE AWS REGION WHERE YOU WANT TO DEPLOY THE ELASTIC SERVERLESS FORWARDER
}

data "aws_serverlessapplicationrepository_application" "esf_sar" {
  application_id = "arn:aws:serverlessrepo:eu-central-1:267093732750:applications/elastic-serverless-forwarder"
}

resource "aws_serverlessapplicationrepository_cloudformation_stack" "esf_cf_stak" {
  name             = "terraform-elastic-serverless-forwarder"
  application_id   = data.aws_serverlessapplicationrepository_application.esf_sar.application_id
  semantic_version = data.aws_serverlessapplicationrepository_application.esf_sar.semantic_version
  capabilities     = data.aws_serverlessapplicationrepository_application.esf_sar.required_capabilities

  parameters = {
    ElasticServerlessForwarderS3ConfigFile         = "s3://elk-config-sar-bucket-dev/sarconfig.yaml"  ## FILL WITH THE VALUE OF THE S3 URL IN THE FORMAT "s3://bucket-name/config-file-name" POINTING TO THE CONFIGURATION FILE FOR YOUR DEPLOYMENT OF THE ELASTIC SERVERLESS FORWARDER
    ElasticServerlessForwarderSSMSecrets           = ""  ## FILL WITH A COMMA DELIMITED LIST OF AWS SSM SECRETS ARNS REFERENCED IN THE CONFIG YAML FILE (IF ANY).
    ElasticServerlessForwarderKMSKeys              = ""  ## FILL WITH A COMMA DELIMITED LIST OF AWS KMS KEYS ARNS TO BE USED FOR DECRYPTING AWS SSM SECRETS REFERENCED IN THE CONFIG YAML FILE (IF ANY).
    ElasticServerlessForwarderSQSEvents            = ""  ## FILL WITH A COMMA DELIMITED LIST OF DIRECT SQS QUEUES ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
    ElasticServerlessForwarderS3SQSEvents          = ""  ## FILL WITH A COMMA DELIMITED LIST OF S3 SQS EVENT NOTIFICATIONS ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
    ElasticServerlessForwarderKinesisEvents        = ""  ## FILL WITH A COMMA DELIMITED LIST OF KINESIS DATA STREAM ARNS TO SET AS EVENT TRIGGERS FOR THE LAMBDA (IF ANY).
    ElasticServerlessForwarderCloudWatchLogsEvents = "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"  ## FILL WITH A COMMA DELIMITED LIST OF CLOUDWATCH LOGS LOG GROUPS ARNS TO SET SUBSCRIPTION FILTERS ON THE LAMBDA FOR (IF ANY).
    ElasticServerlessForwarderS3Buckets            = ""  ## FILL WITH A COMMA DELIMITED LIST OF S3 BUCKETS ARNS THAT ARE THE SOURCES OF THE S3 SQS EVENT NOTIFICATIONS (IF ANY).
  }
}

My sarconfig.yaml

inputs:
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"
    outputs:
      - type: "elasticsearch"
        args:
          # either elasticsearch_url or cloud_id, elasticsearch_url takes precedence
          #elasticsearch_url: "http(s)://domain.tld:port"
          cloud_id: "AWS_test:d********************pbyRmZjBmNDBlNTVmMzY0MWYwYWI2NTg5ODNmN2E4ODllNCRlZmE5NDQ1NWZlZmI0YWIxYWYzOGQ4ZmQ5MDFkOTY3Yw=="
          # either api_key or username/password, api_key takes precedence
          api_key: "******************Tk86WDIxWXg1X1FSaDJfRlJlQnc2c2QyQQ=="
          #username: "username"
          #password: "password"
          es_datastream_name: "test"
          batch_max_actions: 500
          batch_max_bytes: 10485760

Permissions are like this, it was added by deployment, I didn't change it

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "s3:GetObject",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/condo-loan-asyc-dev-check-expired-order:*",
                "arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
            ]
        }
    ]
}

I did a test and released it with permission for everyone:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
                "arn:aws:sqs:us-east-1:*****45454:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:*",
            "Resource": [
                "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*",
                "arn:aws:logs:*:*****45454:log-group:*:log-stream:*",
                "arn:aws:logs:*:*****45454:destination:*",
                "arn:aws:logs:*:*****45454:log-group:*"
            ]
        }
    ]
}

Now we have no error log, but no input yet in Elastic.

{
    "@timestamp": "2022-06-24T06:31:54.515Z",
    "log.level": "warning",
    "message": "no input defined",
    "ecs": {
        "version": "1.6.0"
    },
    "input_id": "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*",
    "input_type": "cloudwatch-logs",
    "log": {
        "logger": "root",
        "origin": {
            "file": {
                "line": 119,
                "name": "handler.py"
            },
            "function": "lambda_handler"
        },
        "original": "no input defined"
    },
    "process": {
        "name": "MainProcess",
        "pid": 9,
        "thread": {
            "id": 139822395262784,
            "name": "MainThread"
        }
    }
}

Andrea_Spacca · June 24, 2022, 6:50am

this should be arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order

that's why the permission was give to the wrong resource:
arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/condo-loan-asyc-dev-check-expired-order:*
vs
arn:aws:logs:us-east-1:*****45454:log-group:*:*

while this solves your current issue it is indeed a bug in the forwarder: the correct ARN format for cloudwatch logs log groups is the one you've provided, not the one expected by the forwarder, I will fix ASAP

Renato_Souza · June 24, 2022, 12:37pm

Ok @Andrea_Spacca , thank you.

After the change the permission was given correctly. The permission issue is ok. But the data is still not sent, is this the BUG?

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::elk-config-sar-bucket-dev/sarconfig.yaml",
            "Effect": "Allow"
        },
        {
            "Action": "logs:DescribeLogGroups",
            "Resource": "arn:aws:logs:us-east-1:*******3299:log-group:*:*",
            "Effect": "Allow"
        },
        {
            "Action": "sqs:SendMessage",
            "Resource": [
                "arn:aws:sqs:us-east-1:*******3299:elastic-serverless-forwarder-replay-queue-128e6a1cd335",
                "arn:aws:sqs:us-east-1:*******3299:elastic-serverless-forwarder-continuing-queue-128e6a1cd335"
            ],
            "Effect": "Allow"
        }
    ]
}

This will happen because the arn is incorrect, because the correct arn has the asterisk, correct?

Andrea_Spacca · June 27, 2022, 12:20am

if you haven't changed the config file from the last you pasted it should work properly (since it includes the trailing :*):

inputs:
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:us-east-1:*****45454:log-group:/aws/lambda/pod-loan-asyc-dev-check-expired-order:*"

Renato_Souza · June 27, 2022, 1:23am

Now it's working. But what I wanted was lambda invocation metrics, errors, memory usage, but this integration doesn't provide this data. Do I have to do anything additional?

This image from asset dashboard AWS Lambda.

I even configured this integration, but it didn't bring data.

Do I need to configure this output somewhere in sarconfig?

Renato_Souza · June 30, 2022, 7:18pm

@Andrea_Spacca @stephenb can you help me, please?

Andrea_Spacca · July 7, 2022, 3:11am

hello @Renato_Souza , sorry for the late reply

there is no serverless solution at the moment for metrics collection, you can collect them using the elastic agent

Renato_Souza · July 7, 2022, 4:11am

Can you use elastic agent in a lambda? Which document should I use? It would be really cool to have invocation lambda metrics. The integration shows this data, I imagine it is possible. If you tell me the way, I'll look for it. Thanks

Andrea_Spacca · July 7, 2022, 5:39am

you cannot run elastic agent in a lambda, you can install it anyway on an ec2 instance through fleet or as a standalone agent

Renato_Souza · July 7, 2022, 11:14am

with this I can recover data from lambda, innovations and etc? In this case, would you use elastic's native integration with aws and fleet?

Andrea_Spacca · July 7, 2022, 11:57am

consider the elastic serverless forwarder as an alternative to elastic agent AWS S3 input (it has also other native AWS inputs indeed), in regards of logs collection

for metrics collections only elastic agent is available

in both cases you should rely o the AWS integrations available in kibana: only installing the assets in the case of the forwarder, adding the full integration in the case of elastic agent

system · July 21, 2022, 11:58am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
FunctionBeat not able to get CloudWatch Logs Beats functionbeat	2	259	August 29, 2023
Functionbeat lambda timeout on aws elasticsearch connection Beats functionbeat	11	2171	June 2, 2020
Lambda created but no data in Elastic Cloud Beats functionbeat	3	1322	January 30, 2019
AWS lambda end of support for Go1.x runtime Beats functionbeat , lambda	4	344	January 9, 2024
Functionbeat unable to export cloudwatch logs to elastic Beats functionbeat	3	430	June 28, 2022

How to integrate AWS Lambda with plugin Elastic

Related topics