[AWS] Functionbeat: no function are enabled for selected provider: 'aws'

stazio · March 28, 2019, 4:03pm

Hi All,

I'm in the process of trying to configure Functionbeat (following the release of ELK 6.7) to ship RDS alert logs, via cloudwatch logs, to my ELK stack.

I have successfully created the S3 bucket and deployed the function to cloudwatch - there is now a 'cloudwatch' subscription on the log group in the Cloudwatch console so I assume everything on that front is working as it should.

I am now trying to run Functionbeat and obtain my first index but am seeing the following error (when running './functionbeat run'):

2019-03-28T15:44:23.277Z	INFO	instance/beat.go:412	functionbeat stopped.
2019-03-28T15:44:23.277Z	ERROR	instance/beat.go:907	Exiting: error when creating the functions, error: no function are enabled for selected provider: 'aws'
Exiting: error when creating the functions, error: no function are enabled for selected provider: 'aws'

My functionbeat.yml file looks like this (stripped down to everything uncommented):

functionbeat.provider.aws.deploy_bucket: "functionbeat-logs"
        functionbeat.provider.aws.functions:
          - name: cloudwatch
        enabled: true
        type: cloudwatch_logs

description: "lambda function for cloudwatch logs"

  - log_group_name: /aws/rds/instance/production-db/alert
    # filter_pattern: mylog_

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]

It seems to me like I am either misunderstanding the command to run the Beat (I have also tried ./functionbeat -e) or the config file thinks that 'enabled: true' is somehow set to false?

I am also seeing the following entry when functionbeat is initialising:

2019-03-28T17:02:50.118Z INFO [functionbeat] beater/functionbeat.go:92 Functionbeat is configuring enabled functions:

The fact that there are no functions listed tells me it's not picking up my 'cloudwatch' function, even though it has been successfully deployed and enabled:true is set.

Many thanks in advance! Any help is greatly appreciated.

stazio · March 29, 2019, 1:04pm

To update this topic, I am also seeing the following messages in the logs:

2019-03-29T12:59:35.281Z INFO [license-manager] licenser/manager.go:265 Cannot retrieve license, retrying later, error: error from server, response code: 0

I am using a Basic ELK license, which I have verified to be valid.
Not sure if this is related to the issue in hand but worth to note for additional information.

When running ./functionbeat, I do however see this in the output which contradicts these messages:

2019-03-29T16:20:33.872Z	INFO	[license-manager]	licenser/manager.go:282	Valid license retrieved	{"license mode": "Basic", "type": "Basic", "status": "Active"}
2019-03-29T16:20:33.872Z	INFO	[functionbeat]	licenser/manager.go:339	License is valid, mode: Basic

Thanks again.

Michal_Pristas · April 1, 2019, 11:48am

Hey @stazio

to me it looks like two instances you're trying to run use different configuration.
First one (faulty one) probably does not have anything configured, no Elasticsearch host to connect to, or even the configuration you are stating in your first post is not visible to the instance.
can you share how exactly you start these two instances and where the configuration you refer to is stored?

Thank you

stazio · April 1, 2019, 12:53pm

Hi Michal,

I only have a single Elasticsearch instance and the Functionbeat service is installed onto the same server that this instance is running on.

I have been using Elasticsearch with Kibana, Logstash, Filebeat and Packetbeat for some time now without any issues - the problem here is strictly with Functionbeat which was supposed to be a new addition to the stack.

All other services were installed via RPM on a Red Hat machine, where I have an Elastic repository configured and hence was able to use 'yum install' for the above, and are started via 'systemctl start [service]', however Functionbeat was downloaded directly from the Elastic website (a .tar.gz package) and I have unpacked it and am running it via the Functionbeat executable.

I have had no issues deploying my function to AWS Lambda and it is successfully connected to my Cloudwatch log group. This was achieved via 'functionbeat deploy cloudwatch', but when executing 'functionbeat run -e' or 'functionbeat -e' I am seeing the error mentioned in the original post.

2019-03-28T15:44:23.277Z	INFO	instance/beat.go:412	functionbeat stopped.
2019-03-28T15:44:23.277Z	ERROR	instance/beat.go:907	Exiting: error when creating the functions, error: no function are enabled for selected provider: 'aws'
Exiting: error when creating the functions, error: no function are enabled for selected provider: 'aws'

I hope that clears things up.

Thanks in advance.

Michal_Pristas · April 1, 2019, 1:33pm

the error mentioned can happen in few scenarios.

first one is, function is missing from a config file.
it looks like this is not the case, but check indentation of the config. in the snipped it looks broken
function is not enabled: the config says it is
function is not deployed: please verify deployment and use -e flag to see the output in the console. Also you can check ENV VAR ENABLED_FUNCTIONS which should contain aws. This env variable contains a list of deployed functions.

stazio · April 1, 2019, 2:35pm

Hi Michal,

Many thanks for your response.
I have verified the indentations - these were off in the original post but that is just how they showed up - they are fine in my config file. I have also used 'functionbeat test' to verify the config and it returns OK. As you have said, the function is indeed enabled in the config.

Please can you let me know how to check the ENABLED_FUNCTIONS environment variable?
I have ran 'echo $ENABLED_FUNCTIONS' which has returned nothing.

Thanks in advance.

Michal_Pristas · April 2, 2019, 11:26am

ok let me just go through it one more time to make sure I understand your situation, I'm getting confused.

you have ES server running
on this ES server you have functionbeat (A) together with any other beats
you also ran functionbeat deploy cloud watch to deploy functionbeat to lambda (B) and set AWS environment variables to correct values (before deploying)

now, beat B (the one deployed to lambda) works as expected
when you try to run beat A, it shows the error mentioned in the first post
Have you tried running ./functionbeat run -e --setup ?
setup will prepare your environment for functionbeat to work properly (will setup dashboards, templates and ML)

stazio · April 2, 2019, 11:43am

That sounds about right, although in your walkthrough just now is beat A not the same beat as beat B?

In my implementation, I have deployed the function 'cloudwatch' via functionbeat by running 'functionbeat deploy cloudwatch' and this was successful on the AWS side, meaning there are no issues with access keys etc.

When running 'functionbeat run -e' or 'functionbeat -e' I get the error mentioned in the first post as you said.

When running 'functionbeat run -e --setup' I get the following error:

2019-04-02T12:41:53.015+0100	ERROR	instance/beat.go:907	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /etc/functionbeat-6.7.0-linux-x86_64/kibana: No dashboards to import. Please make sure the /etc/functionbeat-6.7.0-linux-x86_64/kibana/6 directory contains a dashboard directory.
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /etc/functionbeat-6.7.0-linux-x86_64/kibana: No dashboards to import. Please make sure the /etc/functionbeat-6.7.0-linux-x86_64/kibana/6 directory contains a dashboard directory.

I have checked and there are no default dashboards packaged with Functionbeat (as of yet) so this error makes sense as the setup flag has nothing to execute.

Thanks.

system · April 23, 2019, 1:43am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.