Configuring Functionbeat with AWS VPC information for self-hosted ELK

Good day. I have been working with Functionbeat for the past couple weeks. I was finally able to get logs to my self hosted ELK but I had to modify the IAM role Functionbeat created by adding the following policies:

  • AWSLambdaBasicExecutionRole
  • AWSLambdaVPCAccessExecutionRole

Once added I was then able to add the VPC setting to the created cloudwatch lambda and I had logs showing up in Kibana. Yay!

I would really like to not have to take the manual steps to get this working. Looking further into functionbeat.yml I see that we have the options:

  • role:
  • virtual_private_cloud:

Oh goodie I thought, I can create the role with the needed permissions. I created a policy with the below:

{
   "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogStream",
                "Logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:<AccountNumber>:log-group:/aws/lambda/<beatName>:*"
            ],
            "Effect": "Allow"
        },
	{
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        },
	{
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

The above is the combination of the policies I had to add to the previously created lambda to be able to get logs to my hosted ELK.

Using the above policy on a created Role and using said role in functionbeat.yml I am not getting the same results.

functionbeat-6.7.1-linux-x86_64/functionbeat.yml

functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy"
functionbeat.provider.aws.functions:
  - name: cw-test-application
    enabled: true
    type: cloudwatch_logs
    description: "lambda function for cloudwatch logs"
    role: arn:aws:iam::<AccountNumber>:role/<createdLambdaRole>
    virtual_private_cloud:
      security_group_ids: ["sg-XXXXXXXXXXX"]
      subnet_ids: ["subnet-XXXXXXXXXXX", "subnet-XXXXXXXXXXX"]
    triggers:
      - log_group_name: dev-application
output.elasticsearch:
  hosts: ["10.X.X.X:9200"]

The security group and subnets are what I have working on Functionbeat created Lambda so I know they are good. What do I need to do to make this work without the manual steps? Better yet does anyone have this in terraform yet (I know, asking alot here)?

Thank you in advance.

So follow up (more issues)

Like I said above, I manually created a role with the above policy. Noted the ARN and used that in the role: option in functionbeat.yml.

Unfortunately the expected behavior (use defined ARN) didn't happen. Functionbeat created a new role again. So this might not be baked enough?

I am doing something wrong?

Could you please share the debug logs of deploying the function? Also, please format your configuration using </>.

Thank you for following up. Are you asking for logs/functionbeat?

Also formatted the text as requested (sorry about that).

Exactly. The logs functionbeat prints when starting it with -e -d "*".

Here is the output of running the deploy. For whatever reason the deployment isn't writing to logs/functionbeat but it is there:

root@ip-172-20-3-156:/home/ubuntu/functionbeat-6.7.1-linux-x86_64# ls -lah logs/
total 32K
drwx------ 2 root root 4.0K Apr 30 13:29 .
drwxrwxr-x 6 root root 4.0K Apr 30 13:02 ..
-rw-r--r-- 1 root root 0 Apr 30 13:29 functionbeat
-rw------- 1 root root 4.3K Apr 17 14:21 functionbeat.1
-rw------- 1 root root 4.3K Apr 15 17:35 functionbeat.2
-rw------- 1 root root 4.3K Apr 17 14:32 functionbeat.log

Out put of the deployment:

root@ip-172-0-0-156:/home/ubuntu/functionbeat-6.7.1-linux-x86_64# ./functionbeat --c functionbeat-network.yml -v -e -d "*" deploy cw-test-ntwk-gryphon
2019-04-30T13:33:18.398Z	INFO	instance/beat.go:611	Home path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64] Config path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64] Data path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64/data] Logs path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64/logs]
2019-04-30T13:33:18.398Z	DEBUG	[beat]	instance/beat.go:648	Beat metadata path: /home/ubuntu/functionbeat-6.7.1-linux-x86_64/data/meta.json
2019-04-30T13:33:18.398Z	INFO	instance/beat.go:618	Beat UUID: 4015ad28-1534-4e70-b8fd-4106d882bc8f
2019-04-30T13:33:18.398Z	DEBUG	[cli-handler]	cmd/cli_handler.go:46	Starting deploy for: cw-test-ntwk-gryphon
2019-04-30T13:33:18.398Z	DEBUG	[aws]	aws/cli_manager.go:265	Deploying function: cw-test-ntwk-gryphon
2019-04-30T13:33:18.398Z	DEBUG	[aws]	aws/cli_manager.go:184	Compressing all assets into an artifact
2019-04-30T13:33:18.400Z	DEBUG	[keystore]	keystore/keystore.go:89	Loading file keystore from /home/ubuntu/functionbeat-6.7.1-linux-x86_64/data/functionbeat.keystore
2019-04-30T13:33:20.178Z	DEBUG	[aws]	aws/cli_manager.go:189	Compression is successful (zip size: 15235332 bytes)
2019-04-30T13:33:20.231Z	DEBUG	[aws]	aws/cli_manager.go:215	Using cloudformation template:
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "fnbcwtestntwkgryphon": {
      "Properties": {
        "Code": {
          "S3Bucket": "sos-functionbeat-deploy",
          "S3Key": "functionbeat-deployment/cw-test-ntwk-gryphon/nxRPClsJS61EoyPkY6VbmskylqUm4l0q2rhCxMUrjIY/functionbeat.zip"
        },
        "Description": "lambda function for cloudwatch logs",
        "Environment": {
          "Variables": {
            "BEAT_STRICT_PERMS": "false",
            "ENABLED_FUNCTIONS": "cw-test-ntwk-gryphon"
          }
        },
        "FunctionName": "cw-test-ntwk-gryphon",
        "Handler": "functionbeat",
        "MemorySize": 128,
        "Role": {
          "Fn::GetAtt": [
            "fnbcwtestntwkgryphonIAMRoleLambdaExecution",
            "Arn"
          ]
        },
        "Runtime": "go1.x",
        "Timeout": 3
      },
      "Type": "AWS::Lambda::Function"
    },
    "fnbcwtestntwkgryphonIAMRoleLambdaExecution": {
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": {
                  "Fn::Join": [
                    "",
                    [
                      "lambda.",
                      {
                        "Ref": "AWS::URLSuffix"
                      }
                    ]
                  ]
                }
              }
            }
          ]
        },
        "Path": "/",
        "Policies": [
          {
            "PolicyDocument": {
              "Statement": [
                {
                  "Action": [
                    "logs:CreateLogStream",
                    "Logs:PutLogEvents"
                  ],
                  "Effect": "Allow",
                  "Resource": [
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cw-test-ntwk-gryphon:*"
                    }
                  ]
                }
              ]
            },
            "PolicyName": {
              "Fn::Join": [
                "-",
                [
                  "fnb",
                  "lambda",
                  "cw-test-ntwk-gryphon"
                ]
              ]
            }
          }
        ],
        "RoleName": "functionbeat-lambda-cw-test-ntwk-gryphon"
      },
      "Type": "AWS::IAM::Role"
    },
    "fnbcwtestntwkgryphonLogGroup": {
      "Properties": {
        "LogGroupName": "/aws/lambda/cw-test-ntwk-gryphon"
      },
      "Type": "AWS::Logs::LogGroup"
    },
    "fnbcwtestntwkgryphonPermission0": {
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Fn::GetAtt": [
            "fnbcwtestntwkgryphon",
            "Arn"
          ]
        },
        "Principal": {
          "Fn::Join": [
            "",
            [
              "logs.",
              {
                "Ref": "AWS::Region"
              },
              ".",
              {
                "Ref": "AWS::URLSuffix"
              }
            ]
          ]
        },
        "SourceArn": {
          "Fn::Join": [
            "",
            [
              "arn:",
              {
                "Ref": "AWS::Partition"
              },
              ":logs:",
              {
                "Ref": "AWS::Region"
              },
              ":",
              {
                "Ref": "AWS::AccountId"
              },
              ":log-group:",
              "test-gryphon",
              ":*"
            ]
          ]
        }
      },
      "Type": "AWS::Lambda::Permission"
    },
    "fnbcwtestntwkgryphonSFtestgryphon": {
      "Properties": {
        "DestinationArn": {
          "Fn::GetAtt": [
            "fnbcwtestntwkgryphon",
            "Arn"
          ]
        },
        "FilterPattern": "",
        "LogGroupName": "test-gryphon"
      },
      "Type": "AWS::Logs::SubscriptionFilter"
    }
  }
}

Here is the remainder:

2019-04-30T13:33:20.231Z DEBUG [aws.executor] aws/executor.go:46 The executor is executing '6' operations for converging state
2019-04-30T13:33:20.231Z DEBUG [aws] aws/op_ensure_bucket.go:31 Verifying presence of S3 bucket: sos-functionbeat-deploy
2019-04-30T13:33:20.330Z DEBUG [aws] aws/op_upload_to_bucket.go:42 Uploading file 'functionbeat-deployment/cw-test-ntwk-gryphon/nxRPClsJS61EoyPkY6VbmskylqUm4l0q2rhCxMUrjIY/functionbeat.zip' to bucket 'sos-functionbeat-deploy' with size 15235332 bytes
2019-04-30T13:33:20.753Z DEBUG [aws] aws/op_upload_to_bucket.go:55 Upload successful
2019-04-30T13:33:20.753Z DEBUG [aws] aws/op_upload_to_bucket.go:42 Uploading file 'functionbeat-deployment/cw-test-ntwk-gryphon/-pydLM1zCzWEj9PJ2q9j_ves64eHNwUDfURN9LsmibQ/cloudformation-template-create.json' to bucket 'sos-functionbeat-deploy' with size 4036 bytes
2019-04-30T13:33:20.783Z DEBUG [aws] aws/op_upload_to_bucket.go:55 Upload successful
2019-04-30T13:33:20.783Z DEBUG [aws] aws/op_cloudformation.go:46 Creating CloudFormation create stack request
2019-04-30T13:33:21.246Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-cw-test-ntwk-gryphon-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2019-04-30T13:33:27.323Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbcwtestntwkgryphonLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2019-04-30T13:33:27.323Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbcwtestntwkgryphonIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2019-04-30T13:33:27.323Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbcwtestntwkgryphonLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2019-04-30T13:33:27.323Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbcwtestntwkgryphonIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2019-04-30T13:33:27.323Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbcwtestntwkgryphonLogGroup, ResourceStatus: CREATE_COMPLETE
2019-04-30T13:33:41.517Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbcwtestntwkgryphonIAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2019-04-30T13:33:45.574Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbcwtestntwkgryphon, ResourceStatus: CREATE_IN_PROGRESS
2019-04-30T13:33:45.574Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbcwtestntwkgryphon, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2019-04-30T13:33:47.609Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbcwtestntwkgryphon, ResourceStatus: CREATE_COMPLETE
2019-04-30T13:33:49.646Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcwtestntwkgryphonSFtestgryphon, ResourceStatus: CREATE_IN_PROGRESS
2019-04-30T13:33:49.646Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbcwtestntwkgryphonPermission0, ResourceStatus: CREATE_IN_PROGRESS
2019-04-30T13:33:49.646Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbcwtestntwkgryphonPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2019-04-30T13:33:53.710Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcwtestntwkgryphonSFtestgryphon, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2019-04-30T13:33:53.710Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcwtestntwkgryphonSFtestgryphon, ResourceStatus: CREATE_COMPLETE
2019-04-30T13:33:59.810Z INFO [aws] aws/op_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbcwtestntwkgryphonPermission0, ResourceStatus: CREATE_COMPLETE
2019-04-30T13:34:01.769Z DEBUG [aws] aws/op_delete_file_bucket.go:35 Removing file 'functionbeat-deployment/cw-test-ntwk-gryphon/nxRPClsJS61EoyPkY6VbmskylqUm4l0q2rhCxMUrjIY/functionbeat.zip' on bucket 'sos-functionbeat-deploy'
2019-04-30T13:34:01.882Z DEBUG [aws] aws/op_delete_file_bucket.go:48 Remove successful
2019-04-30T13:34:01.882Z DEBUG [aws.executor] aws/executor.go:61 All operations successful
2019-04-30T13:34:01.882Z DEBUG [aws] aws/cli_manager.go:271 Successfully created function 'cw-test-ntwk-gryphon'
2019-04-30T13:34:01.882Z DEBUG [aws] aws/cli_manager.go:272 Deploy finish for function 'cw-test-ntwk-gryphon'
Function: cw-test-ntwk-gryphon, deploy successful
2019-04-30T13:34:01.882Z DEBUG [cli-handler] cmd/cli_handler.go:66 Deploy execution ended

Checking the new Lambda function, it doesn't appear to use or honor the specified role. It creates a new role again:

functionbeat-lambda-cw-test-ntwk-gryphon

Is your functionbeat version is 6.7.1? The feature hasn't been released yet. You can only try it on master.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.