# Configuring Functionbeat with AWS VPC information for self-hosted ELK

**URL:** <https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508>\
**Category:** Beats\
**Tags:** functionbeat\
**Created:** [April 25, 2019, 4:39pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508 "2019-04-25T16:39:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrewjkrull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewjkrull/32/45041_2.png) [@andrewjkrull](https://discuss.elastic.co/u/andrewjkrull)\
**Post date:** [April 25, 2019, 4:39pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/1 "2019-04-25T16:39:09Z")

</div>

Good day. I have been working with Functionbeat for the past couple weeks. I was finally able to get logs to my self hosted ELK but I had to modify the IAM role Functionbeat created by adding the following policies:

- AWSLambdaBasicExecutionRole
- AWSLambdaVPCAccessExecutionRole

Once added I was then able to add the VPC setting to the created cloudwatch lambda and I had logs showing up in Kibana. Yay!

I would really like to not have to take the manual steps to get this working. Looking further into functionbeat.yml I see that we have the options:

- role:
- virtual\_private\_cloud:

Oh goodie I thought, I can create the role with the needed permissions. I created a policy with the below:

```
{
   "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogStream",
                "Logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:<AccountNumber>:log-group:/aws/lambda/<beatName>:*"
            ],
            "Effect": "Allow"
        },
	{
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        },
	{
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

```

The above is the combination of the policies I had to add to the previously created lambda to be able to get logs to my hosted ELK.

Using the above policy on a created Role and using said role in functionbeat.yml I am not getting the same results.

functionbeat-6.7.1-linux-x86\_64/functionbeat.yml

```
functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy"
functionbeat.provider.aws.functions:
  - name: cw-test-application
    enabled: true
    type: cloudwatch_logs
    description: "lambda function for cloudwatch logs"
    role: arn:aws:iam::<AccountNumber>:role/<createdLambdaRole>
    virtual_private_cloud:
      security_group_ids: ["sg-XXXXXXXXXXX"]
      subnet_ids: ["subnet-XXXXXXXXXXX", "subnet-XXXXXXXXXXX"]
    triggers:
      - log_group_name: dev-application
output.elasticsearch:
  hosts: ["10.X.X.X:9200"]

```

The security group and subnets are what I have working on Functionbeat created Lambda so I know they are good. What do I need to do to make this work without the manual steps? Better yet does anyone have this in terraform yet (I know, asking alot here)?

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![andrewjkrull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewjkrull/32/45041_2.png) [@andrewjkrull](https://discuss.elastic.co/u/andrewjkrull)\
**Post date:** [April 25, 2019, 5:02pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/2 "2019-04-25T17:02:13Z")

</div>

So follow up (more issues)

Like I said above, I manually created a role with the above policy. Noted the ARN and used that in the role: option in functionbeat.yml.

Unfortunately the expected behavior (use defined ARN) didn't happen. Functionbeat created a new role again. So this might not be baked enough?

I am doing something wrong?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 26, 2019, 1:03pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/3 "2019-04-26T13:03:12Z")

</div>

Could you please share the debug logs of deploying the function? Also, please format your configuration using `</>`.

---

<div class="post-metadata">

**Author:** ![andrewjkrull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewjkrull/32/45041_2.png) [@andrewjkrull](https://discuss.elastic.co/u/andrewjkrull)\
**Post date:** [April 29, 2019, 2:18pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/4 "2019-04-29T14:18:26Z")

</div>

Thank you for following up. Are you asking for logs/functionbeat?

Also formatted the text as requested (sorry about that).

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 30, 2019, 6:20am UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/5 "2019-04-30T06:20:19Z")

</div>

Exactly. The logs functionbeat prints when starting it with `-e -d "*"`.

---

<div class="post-metadata">

**Author:** ![andrewjkrull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewjkrull/32/45041_2.png) [@andrewjkrull](https://discuss.elastic.co/u/andrewjkrull)\
**Post date:** [April 30, 2019, 1:42pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/6 "2019-04-30T13:42:06Z")

</div>

Here is the output of running the deploy. For whatever reason the deployment isn't writing to logs/functionbeat but it is there:

root@ip-172-20-3-156:/home/ubuntu/functionbeat-6.7.1-linux-x86\_64# ls -lah logs/  
total 32K  
drwx------ 2 root root 4.0K Apr 30 13:29 .  
drwxrwxr-x 6 root root 4.0K Apr 30 13:02 ..  
-rw-r--r-- 1 root root 0 Apr 30 13:29 functionbeat  
-rw------- 1 root root 4.3K Apr 17 14:21 functionbeat.1  
-rw------- 1 root root 4.3K Apr 15 17:35 functionbeat.2  
-rw------- 1 root root 4.3K Apr 17 14:32 functionbeat.log

Out put of the deployment:

```
root@ip-172-0-0-156:/home/ubuntu/functionbeat-6.7.1-linux-x86_64# ./functionbeat --c functionbeat-network.yml -v -e -d "*" deploy cw-test-ntwk-gryphon
2019-04-30T13:33:18.398Z	INFO	instance/beat.go:611	Home path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64] Config path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64] Data path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64/data] Logs path: [/home/ubuntu/functionbeat-6.7.1-linux-x86_64/logs]
2019-04-30T13:33:18.398Z	DEBUG	[beat]	instance/beat.go:648	Beat metadata path: /home/ubuntu/functionbeat-6.7.1-linux-x86_64/data/meta.json
2019-04-30T13:33:18.398Z	INFO	instance/beat.go:618	Beat UUID: 4015ad28-1534-4e70-b8fd-4106d882bc8f
2019-04-30T13:33:18.398Z	DEBUG	[cli-handler]	cmd/cli_handler.go:46	Starting deploy for: cw-test-ntwk-gryphon
2019-04-30T13:33:18.398Z	DEBUG	[aws]	aws/cli_manager.go:265	Deploying function: cw-test-ntwk-gryphon
2019-04-30T13:33:18.398Z	DEBUG	[aws]	aws/cli_manager.go:184	Compressing all assets into an artifact
2019-04-30T13:33:18.400Z	DEBUG	[keystore]	keystore/keystore.go:89	Loading file keystore from /home/ubuntu/functionbeat-6.7.1-linux-x86_64/data/functionbeat.keystore
2019-04-30T13:33:20.178Z	DEBUG	[aws]	aws/cli_manager.go:189	Compression is successful (zip size: 15235332 bytes)
2019-04-30T13:33:20.231Z	DEBUG	[aws]	aws/cli_manager.go:215	Using cloudformation template:
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "fnbcwtestntwkgryphon": {
      "Properties": {
        "Code": {
          "S3Bucket": "sos-functionbeat-deploy",
          "S3Key": "functionbeat-deployment/cw-test-ntwk-gryphon/nxRPClsJS61EoyPkY6VbmskylqUm4l0q2rhCxMUrjIY/functionbeat.zip"
        },
        "Description": "lambda function for cloudwatch logs",
        "Environment": {
          "Variables": {
            "BEAT_STRICT_PERMS": "false",
            "ENABLED_FUNCTIONS": "cw-test-ntwk-gryphon"
          }
        },
        "FunctionName": "cw-test-ntwk-gryphon",
        "Handler": "functionbeat",
        "MemorySize": 128,
        "Role": {
          "Fn::GetAtt": [
            "fnbcwtestntwkgryphonIAMRoleLambdaExecution",
            "Arn"
          ]
        },
        "Runtime": "go1.x",
        "Timeout": 3
      },
      "Type": "AWS::Lambda::Function"
    },
    "fnbcwtestntwkgryphonIAMRoleLambdaExecution": {
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": {
                  "Fn::Join": [
                    "",
                    [
                      "lambda.",
                      {
                        "Ref": "AWS::URLSuffix"
                      }
                    ]
                  ]
                }
              }
            }
          ]
        },
        "Path": "/",
        "Policies": [
          {
            "PolicyDocument": {
              "Statement": [
                {
                  "Action": [
                    "logs:CreateLogStream",
                    "Logs:PutLogEvents"
                  ],
                  "Effect": "Allow",
                  "Resource": [
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cw-test-ntwk-gryphon:*"
                    }
                  ]
                }
              ]
            },
            "PolicyName": {
              "Fn::Join": [
                "-",
                [
                  "fnb",
                  "lambda",
                  "cw-test-ntwk-gryphon"
                ]
              ]
            }
          }
        ],
        "RoleName": "functionbeat-lambda-cw-test-ntwk-gryphon"
      },
      "Type": "AWS::IAM::Role"
    },
    "fnbcwtestntwkgryphonLogGroup": {
      "Properties": {
        "LogGroupName": "/aws/lambda/cw-test-ntwk-gryphon"
      },
      "Type": "AWS::Logs::LogGroup"
    },
    "fnbcwtestntwkgryphonPermission0": {
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Fn::GetAtt": [
            "fnbcwtestntwkgryphon",
            "Arn"
          ]
        },
        "Principal": {
          "Fn::Join": [
            "",
            [
              "logs.",
              {
                "Ref": "AWS::Region"
              },
              ".",
              {
                "Ref": "AWS::URLSuffix"
              }
            ]
          ]
        },
        "SourceArn": {
          "Fn::Join": [
            "",
            [
              "arn:",
              {
                "Ref": "AWS::Partition"
              },
              ":logs:",
              {
                "Ref": "AWS::Region"
              },
              ":",
              {
                "Ref": "AWS::AccountId"
              },
              ":log-group:",
              "test-gryphon",
              ":*"
            ]
          ]
        }
      },
      "Type": "AWS::Lambda::Permission"
    },
    "fnbcwtestntwkgryphonSFtestgryphon": {
      "Properties": {
        "DestinationArn": {
          "Fn::GetAtt": [
            "fnbcwtestntwkgryphon",
            "Arn"
          ]
        },
        "FilterPattern": "",
        "LogGroupName": "test-gryphon"
      },
      "Type": "AWS::Logs::SubscriptionFilter"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![andrewjkrull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewjkrull/32/45041_2.png) [@andrewjkrull](https://discuss.elastic.co/u/andrewjkrull)\
**Post date:** [April 30, 2019, 1:42pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/7 "2019-04-30T13:42:51Z")

</div>

Here is the remainder:

2019-04-30T13:33:20.231Z DEBUG [aws.executor] aws/executor.go:46 The executor is executing '6' operations for converging state  
2019-04-30T13:33:20.231Z DEBUG [aws] aws/op\_ensure\_bucket.go:31 Verifying presence of S3 bucket: sos-functionbeat-deploy  
2019-04-30T13:33:20.330Z DEBUG [aws] aws/op\_upload\_to\_bucket.go:42 Uploading file 'functionbeat-deployment/cw-test-ntwk-gryphon/nxRPClsJS61EoyPkY6VbmskylqUm4l0q2rhCxMUrjIY/functionbeat.zip' to bucket 'sos-functionbeat-deploy' with size 15235332 bytes  
2019-04-30T13:33:20.753Z DEBUG [aws] aws/op\_upload\_to\_bucket.go:55 Upload successful  
2019-04-30T13:33:20.753Z DEBUG [aws] aws/op\_upload\_to\_bucket.go:42 Uploading file 'functionbeat-deployment/cw-test-ntwk-gryphon/-pydLM1zCzWEj9PJ2q9j\_ves64eHNwUDfURN9LsmibQ/cloudformation-template-create.json' to bucket 'sos-functionbeat-deploy' with size 4036 bytes  
2019-04-30T13:33:20.783Z DEBUG [aws] aws/op\_upload\_to\_bucket.go:55 Upload successful  
2019-04-30T13:33:20.783Z DEBUG [aws] aws/op\_cloudformation.go:46 Creating CloudFormation create stack request  
2019-04-30T13:33:21.246Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-cw-test-ntwk-gryphon-stack, ResourceStatus: CREATE\_IN\_PROGRESS, ResourceStatusReason: User Initiated  
2019-04-30T13:33:27.323Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbcwtestntwkgryphonLogGroup, ResourceStatus: CREATE\_IN\_PROGRESS  
2019-04-30T13:33:27.323Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbcwtestntwkgryphonIAMRoleLambdaExecution, ResourceStatus: CREATE\_IN\_PROGRESS  
2019-04-30T13:33:27.323Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbcwtestntwkgryphonLogGroup, ResourceStatus: CREATE\_IN\_PROGRESS, ResourceStatusReason: Resource creation Initiated  
2019-04-30T13:33:27.323Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbcwtestntwkgryphonIAMRoleLambdaExecution, ResourceStatus: CREATE\_IN\_PROGRESS, ResourceStatusReason: Resource creation Initiated  
2019-04-30T13:33:27.323Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbcwtestntwkgryphonLogGroup, ResourceStatus: CREATE\_COMPLETE  
2019-04-30T13:33:41.517Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbcwtestntwkgryphonIAMRoleLambdaExecution, ResourceStatus: CREATE\_COMPLETE  
2019-04-30T13:33:45.574Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbcwtestntwkgryphon, ResourceStatus: CREATE\_IN\_PROGRESS  
2019-04-30T13:33:45.574Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbcwtestntwkgryphon, ResourceStatus: CREATE\_IN\_PROGRESS, ResourceStatusReason: Resource creation Initiated  
2019-04-30T13:33:47.609Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbcwtestntwkgryphon, ResourceStatus: CREATE\_COMPLETE  
2019-04-30T13:33:49.646Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcwtestntwkgryphonSFtestgryphon, ResourceStatus: CREATE\_IN\_PROGRESS  
2019-04-30T13:33:49.646Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbcwtestntwkgryphonPermission0, ResourceStatus: CREATE\_IN\_PROGRESS  
2019-04-30T13:33:49.646Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbcwtestntwkgryphonPermission0, ResourceStatus: CREATE\_IN\_PROGRESS, ResourceStatusReason: Resource creation Initiated  
2019-04-30T13:33:53.710Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcwtestntwkgryphonSFtestgryphon, ResourceStatus: CREATE\_IN\_PROGRESS, ResourceStatusReason: Resource creation Initiated  
2019-04-30T13:33:53.710Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcwtestntwkgryphonSFtestgryphon, ResourceStatus: CREATE\_COMPLETE  
2019-04-30T13:33:59.810Z INFO [aws] aws/op\_cloudformation.go:95 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbcwtestntwkgryphonPermission0, ResourceStatus: CREATE\_COMPLETE  
2019-04-30T13:34:01.769Z DEBUG [aws] aws/op\_delete\_file\_bucket.go:35 Removing file 'functionbeat-deployment/cw-test-ntwk-gryphon/nxRPClsJS61EoyPkY6VbmskylqUm4l0q2rhCxMUrjIY/functionbeat.zip' on bucket 'sos-functionbeat-deploy'  
2019-04-30T13:34:01.882Z DEBUG [aws] aws/op\_delete\_file\_bucket.go:48 Remove successful  
2019-04-30T13:34:01.882Z DEBUG [aws.executor] aws/executor.go:61 All operations successful  
2019-04-30T13:34:01.882Z DEBUG [aws] aws/cli\_manager.go:271 Successfully created function 'cw-test-ntwk-gryphon'  
2019-04-30T13:34:01.882Z DEBUG [aws] aws/cli\_manager.go:272 Deploy finish for function 'cw-test-ntwk-gryphon'  
Function: cw-test-ntwk-gryphon, deploy successful  
2019-04-30T13:34:01.882Z DEBUG [cli-handler] cmd/cli\_handler.go:66 Deploy execution ended

---

<div class="post-metadata">

**Author:** ![andrewjkrull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewjkrull/32/45041_2.png) [@andrewjkrull](https://discuss.elastic.co/u/andrewjkrull)\
**Post date:** [April 30, 2019, 1:45pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/8 "2019-04-30T13:45:09Z")

</div>

Checking the new Lambda function, it doesn't appear to use or honor the specified role. It creates a new role again:

functionbeat-lambda-cw-test-ntwk-gryphon

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 6, 2019, 7:19am UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/9 "2019-05-06T07:19:23Z")

</div>

Is your functionbeat version is 6.7.1? The feature hasn't been released yet. You can only try it on master.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 26, 2019, 9:19pm UTC](https://discuss.elastic.co/t/configuring-functionbeat-with-aws-vpc-information-for-self-hosted-elk/178508/10 "2019-05-26T21:19:25Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.